Nov 01 12:19:21.739685 ip-10-0-140-174 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 12:19:21.739700 ip-10-0-140-174 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 12:19:21.739710 ip-10-0-140-174 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 12:19:21.740028 ip-10-0-140-174 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 12:19:31.842700 ip-10-0-140-174 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 12:19:31.842718 ip-10-0-140-174 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 526e5d3e0ef3402cb16cbe243405f14b -- Nov 01 12:21:30.254576 ip-10-0-140-174 systemd[1]: Starting Kubernetes Kubelet... Nov 01 12:21:30.706251 ip-10-0-140-174 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 12:21:30.706251 ip-10-0-140-174 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 12:21:30.706251 ip-10-0-140-174 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 12:21:30.706251 ip-10-0-140-174 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 12:21:30.709827 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709750 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.709827 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709827 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709831 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709835 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709838 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709841 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709844 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709847 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709850 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709853 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709857 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709861 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709865 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709869 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709872 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709875 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709878 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709881 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709883 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709886 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.709896 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709889 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709892 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709894 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709897 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709900 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709905 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709908 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709911 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709914 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709916 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709919 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709928 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709931 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709933 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709936 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709938 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709941 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709943 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.710344 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709946 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709948 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709951 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709953 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709956 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709958 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709961 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709964 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709966 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709969 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709971 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709974 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709977 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709980 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709982 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709985 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709988 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709990 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709993 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709995 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.710790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.709998 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710000 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710003 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710005 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710008 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710010 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710020 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710023 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710025 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710028 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710030 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710033 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710036 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710038 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710041 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710043 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710046 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710049 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710052 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.711261 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710054 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710056 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710059 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710062 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710064 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710069 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710073 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710076 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710078 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710081 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710084 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710086 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710089 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710092 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710095 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710097 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710100 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710102 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710105 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.711727 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710108 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710116 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710118 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710122 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710125 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710127 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710130 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710132 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.710135 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711020 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711027 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711029 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711033 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711035 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711038 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711041 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711044 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711047 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711049 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711052 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.712250 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711054 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711057 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711068 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711071 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711074 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711077 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711079 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711082 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711085 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711087 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711090 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711093 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711095 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711098 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711107 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711109 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711112 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711115 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711118 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.712755 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711122 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711124 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711127 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711130 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711132 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711135 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711137 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711140 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711143 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711145 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711148 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711151 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711153 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711156 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711159 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711161 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711164 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711169 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711172 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.713397 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711175 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711178 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711181 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711183 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711186 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711189 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711191 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711194 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711197 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711205 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711208 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711210 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711213 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711215 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711218 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711220 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711223 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711225 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711228 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711230 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.713926 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711233 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711236 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711239 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711241 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711244 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711246 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711249 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711252 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711254 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711257 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711259 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711262 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711265 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711267 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711270 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711273 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711275 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711278 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711280 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711282 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.714415 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711286 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711288 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711296 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711312 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711315 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711318 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711321 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711323 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711326 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711328 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711330 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711333 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711336 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711339 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711341 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711344 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711346 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711852 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711857 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.714973 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711860 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711862 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711865 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711868 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711871 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711873 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711876 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711878 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711881 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711884 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711886 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711889 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711892 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711895 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711898 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711901 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711903 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711906 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711909 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711912 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.715454 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711914 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711917 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711919 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711921 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711924 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711927 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711930 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711933 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711935 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711938 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711940 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711943 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711945 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711948 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711950 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711953 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711956 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711958 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711961 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711963 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.716010 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711966 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711968 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711971 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711974 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711979 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711983 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711987 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711991 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711994 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.711997 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712000 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712002 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712005 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712010 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712016 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712018 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712022 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.716493 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712026 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712029 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712032 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712035 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712038 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712041 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712043 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712046 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712048 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712051 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712053 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712056 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712058 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712061 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712063 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712066 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712068 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712071 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712073 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712076 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.716918 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712079 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712081 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712085 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712088 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712091 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712093 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712096 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712100 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712102 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712105 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712107 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712110 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712112 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712115 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712118 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712120 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712123 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712125 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712127 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712130 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.717427 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712133 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712135 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712138 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712140 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712143 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.712145 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712249 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712257 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712262 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712267 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712271 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712275 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712279 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712283 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712286 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712289 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712293 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712310 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712314 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712318 2580 flags.go:64] FLAG: --cgroup-root="" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712321 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712324 2580 flags.go:64] FLAG: --client-ca-file="" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712327 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712330 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 12:21:30.717898 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712334 2580 flags.go:64] FLAG: --cluster-domain="" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712337 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712340 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712343 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712347 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712351 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712354 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712357 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712360 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712363 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712366 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712369 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712372 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712375 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712379 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712382 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712385 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712388 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712391 2580 flags.go:64] FLAG: --enable-server="true" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712396 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712400 2580 flags.go:64] FLAG: --event-burst="100" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712403 2580 flags.go:64] FLAG: --event-qps="50" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712406 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712409 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 12:21:30.718481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712412 2580 flags.go:64] FLAG: --eviction-hard="" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712418 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712421 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712424 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712427 2580 flags.go:64] FLAG: --eviction-soft="" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712430 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712433 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712436 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712439 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712442 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712445 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712448 2580 flags.go:64] FLAG: --feature-gates="" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712452 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712455 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712458 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712462 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712464 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712467 2580 flags.go:64] FLAG: --help="false" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712471 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712475 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712478 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712480 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712484 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 12:21:30.719066 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712487 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712490 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712493 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712496 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712500 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712503 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712506 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712510 2580 flags.go:64] FLAG: --kube-reserved="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712513 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712516 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712519 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712522 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712525 2580 flags.go:64] FLAG: --lock-file="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712528 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712531 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712534 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712540 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712543 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712545 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712548 2580 flags.go:64] FLAG: --logging-format="text" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712551 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712555 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712558 2580 flags.go:64] FLAG: --manifest-url="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712560 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712564 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 12:21:30.719639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712567 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712572 2580 flags.go:64] FLAG: --max-pods="110" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712575 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712578 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712580 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712583 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712586 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712589 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712592 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712599 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712602 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712606 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712609 2580 flags.go:64] FLAG: --pod-cidr="" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712612 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712615 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712618 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712621 2580 flags.go:64] FLAG: --port="10250" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712624 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712627 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08eb0fe37f3bce0c7" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712631 2580 flags.go:64] FLAG: --qos-reserved="" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712634 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712637 2580 flags.go:64] FLAG: --register-node="true" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712640 2580 flags.go:64] FLAG: --register-with-taints="" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712644 2580 flags.go:64] FLAG: --registry-burst="10" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712646 2580 flags.go:64] FLAG: --registry-qps="5" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712649 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 01 12:21:30.720222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712652 2580 flags.go:64] FLAG: --reserved-memory="" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712656 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712659 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712662 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712665 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712668 2580 flags.go:64] FLAG: --runonce="false" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712670 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712673 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712676 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712679 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712682 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712685 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712688 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712691 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712694 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712697 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712700 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712704 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712708 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712711 2580 flags.go:64] FLAG: --system-cgroups="" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712714 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712719 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712722 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712725 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712730 2580 flags.go:64] FLAG: --tls-min-version="" Nov 01 12:21:30.720886 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712733 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712736 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712739 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712742 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712745 2580 flags.go:64] FLAG: --v="2" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712749 2580 flags.go:64] FLAG: --version="false" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712753 2580 flags.go:64] FLAG: --vmodule="" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712757 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.712761 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714885 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714910 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714916 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714920 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714927 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714932 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714936 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714940 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714945 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714949 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714954 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714958 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.721480 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714962 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714966 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714975 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714980 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714985 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.714990 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715004 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715008 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715025 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715028 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715154 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715162 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715165 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715168 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715170 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715173 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715176 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715178 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715181 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715184 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.721985 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715187 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715192 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715195 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715197 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715200 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715203 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715205 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715208 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715210 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715213 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715216 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715219 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715222 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715224 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715227 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715230 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715232 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715235 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715237 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.722469 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715240 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715242 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715245 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715247 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715250 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715253 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715256 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715259 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715262 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715265 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715267 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715270 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715273 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715275 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715279 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715282 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715284 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715287 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715290 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.722917 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715293 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715295 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715310 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715314 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715316 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715321 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715324 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715327 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715331 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715334 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715336 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715339 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715342 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715347 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715350 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715352 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715355 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715358 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715362 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.723378 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715367 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715372 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715376 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715378 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715381 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715384 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715386 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715389 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715395 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715397 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715400 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715403 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715405 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715408 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715411 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715413 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.723820 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715416 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.715449 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715499 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715503 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715506 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715509 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715512 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715515 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715519 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715521 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715524 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715527 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715530 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715533 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715535 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715538 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.724186 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715540 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715543 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715545 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715548 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715551 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715553 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715556 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715559 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715561 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715565 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715569 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715573 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715578 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715581 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715584 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715587 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715590 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715592 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715595 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.724580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715598 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715600 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715603 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715605 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715608 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715611 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715614 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715616 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715619 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715622 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715625 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715627 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715630 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715633 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715635 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715638 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715641 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715643 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715647 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.725034 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715649 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715652 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715655 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715659 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715662 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715664 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715667 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715669 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715672 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715675 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715677 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715680 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715683 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715685 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715688 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715692 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715694 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715697 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715700 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715702 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.725484 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715705 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715707 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715713 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715716 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715718 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715721 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715723 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715726 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715728 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715731 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715733 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715736 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715739 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715741 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715745 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715748 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715750 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715753 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.725988 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715755 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715758 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715761 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715763 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715768 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715771 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715774 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715778 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715781 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715783 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715786 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715790 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715792 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.715795 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.723107 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.723121 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.723997 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.724003 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 12:21:30.726539 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724060 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724065 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724068 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724071 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724075 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724078 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724080 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724083 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724085 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724088 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724091 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724093 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724096 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724098 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724101 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724103 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724106 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724109 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724111 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724114 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.726977 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724117 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724120 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724122 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724125 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724128 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724130 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724133 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724136 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724138 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724142 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724145 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724148 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724151 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724153 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724156 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724158 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724161 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724163 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724166 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.727458 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724168 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724171 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724174 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724176 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724178 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724181 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724184 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724187 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724190 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724192 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724195 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724197 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724200 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724203 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724205 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724208 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724210 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724213 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724215 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.727900 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724218 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724220 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724223 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724226 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724230 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724235 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724238 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724241 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724244 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724246 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724249 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724251 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724254 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724257 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724259 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724262 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724264 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724267 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724269 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724272 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.728351 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724274 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724277 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724280 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724283 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724285 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724288 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724291 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724294 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724309 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724313 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724316 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724318 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724321 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724323 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724326 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724328 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724331 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724334 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724337 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724339 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.728814 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724342 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724345 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724347 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724349 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724352 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724355 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724357 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724361 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.724388 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724523 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724528 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724532 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724535 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724537 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724540 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.729276 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724543 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724546 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724548 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724551 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724554 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724558 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724562 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724565 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724568 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724571 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724573 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724576 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724578 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724581 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724584 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724587 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724591 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724593 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724596 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.729639 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724599 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724601 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724604 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724606 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724609 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724611 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724614 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724616 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724619 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724622 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724624 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724626 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724629 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724631 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724634 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724637 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724640 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724644 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724647 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.730087 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724650 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724652 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724655 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724657 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724660 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724662 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724665 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724668 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724670 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724673 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724675 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724678 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724681 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724684 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724686 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724689 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724692 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724694 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724697 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724700 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.730580 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724702 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724705 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724707 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724712 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724715 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724717 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724720 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724722 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724725 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724728 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724731 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724733 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724736 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724738 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724741 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724743 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724746 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724748 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724751 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.731037 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724754 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724756 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724759 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724761 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724764 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724767 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724770 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724772 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724774 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724777 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724779 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724782 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724784 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724787 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724789 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724792 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724794 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724797 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724799 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.731515 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724802 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724804 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724807 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.724830 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724859 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724864 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724867 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724870 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724873 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724875 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724878 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724882 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724885 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724887 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724890 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724893 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:21:30.731969 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724895 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724898 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724901 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724904 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724906 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724909 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724911 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724914 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724916 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724919 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724921 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724924 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724927 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724931 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724934 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724937 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724940 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724943 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724945 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:21:30.732375 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724948 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724951 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724953 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724956 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724958 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724961 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724963 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724966 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724970 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724972 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724975 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724977 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724980 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724983 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724985 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724988 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724991 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724994 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724997 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.724999 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:21:30.732826 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725002 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725004 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725007 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725009 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725012 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725016 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725019 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725022 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725024 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725027 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725029 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725032 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725034 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725037 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725040 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725042 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725045 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725048 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725050 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725053 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:21:30.733281 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725057 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725059 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725062 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725064 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725067 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725070 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725072 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725075 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725077 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725080 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725082 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725085 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725087 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725090 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725094 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725097 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725099 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725102 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725105 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:21:30.733775 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725107 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725110 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725112 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725115 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725117 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725120 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725122 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725125 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725127 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725130 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725134 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725137 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725141 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725145 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725148 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:30.725150 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.725198 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.727465 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 12:21:30.734217 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.728396 2580 server.go:1014] "Starting client certificate rotation" Nov 01 12:21:30.734728 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.728498 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 12:21:30.734728 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.728538 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 12:21:30.734728 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.731352 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 12:21:30.734728 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.732508 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 12:21:30.744583 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.744565 2580 log.go:25] "Validated CRI v1 runtime API" Nov 01 12:21:30.748603 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.748589 2580 log.go:25] "Validated CRI v1 image API" Nov 01 12:21:30.749395 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.749374 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 12:21:30.754638 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.754609 2580 fs.go:135] Filesystem UUIDs: map[6f1500bb-a167-4d5b-951f-4f6b67f05cf3:/dev/nvme0n1p3 755deffa-cdf4-4566-b077-7b875954c767:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 01 12:21:30.754760 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.754634 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 12:21:30.761251 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.761128 2580 manager.go:217] Machine: {Timestamp:2026-11-01 12:21:30.759758754 +0000 UTC m=+0.377084090 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096069 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d54b96f7c04ca6d40b32dfd16fd5a SystemUUID:ec2d54b9-6f7c-04ca-6d40-b32dfd16fd5a BootID:526e5d3e-0ef3-402c-b16c-be243405f14b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ef:b4:85:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ef:b4:85:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:27:98:37:d9:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 12:21:30.761923 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.761909 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 12:21:30.762059 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.762043 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 12:21:30.765801 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.765777 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 12:21:30.765964 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.765803 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-174.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 12:21:30.766043 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.765969 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 12:21:30.766043 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.765981 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 12:21:30.766043 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.765996 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 12:21:30.767777 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.767763 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 12:21:30.769019 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.769006 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 12:21:30.774242 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.774227 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 12:21:30.774394 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.774383 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 12:21:30.775336 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.775325 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 01 12:21:30.775392 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.775343 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 12:21:30.775392 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.775358 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 12:21:30.775392 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.775371 2580 kubelet.go:405] "Adding apiserver pod source" Nov 01 12:21:30.775392 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.775383 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 12:21:30.775717 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.775704 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 12:21:30.778144 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.778128 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 12:21:30.784654 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.784641 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 12:21:30.785532 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785518 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 12:21:30.785592 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785547 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 12:21:30.785698 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785686 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 12:21:30.785733 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785708 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 12:21:30.785733 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785715 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 12:21:30.785733 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785721 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 12:21:30.785733 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785727 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 12:21:30.785733 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.785734 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 12:21:30.786548 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.786539 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 12:21:30.786580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.786548 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 12:21:30.786580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.786556 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 12:21:30.786580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.786561 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 12:21:30.786580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.786575 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 12:21:30.786699 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.786584 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 12:21:30.788329 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.788319 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 12:21:30.788329 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.788330 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 12:21:30.792048 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.792033 2580 server.go:1263] "Started kubelet" Nov 01 12:21:30.792160 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.792137 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 12:21:30.792220 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.792182 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 12:21:30.792261 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.792234 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 12:21:30.793474 ip-10-0-140-174 systemd[1]: Started Kubernetes Kubelet. Nov 01 12:21:30.794511 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.794001 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 12:21:30.795578 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.795564 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 01 12:21:30.800089 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.799958 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 12:21:30.803890 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.803874 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 12:21:30.804542 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.804530 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 12:21:30.804686 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.804666 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" Nov 01 12:21:30.805172 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805153 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 12:21:30.805407 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805292 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 12:21:30.805407 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805407 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 12:21:30.805489 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805409 2580 factory.go:153] Registering CRI-O factory Nov 01 12:21:30.805489 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805422 2580 factory.go:223] Registration of the crio container factory successfully Nov 01 12:21:30.805489 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805466 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 12:21:30.805489 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805476 2580 factory.go:55] Registering systemd factory Nov 01 12:21:30.805489 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805484 2580 factory.go:223] Registration of the systemd container factory successfully Nov 01 12:21:30.805691 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805498 2580 factory.go:103] Registering Raw factory Nov 01 12:21:30.805691 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805476 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 01 12:21:30.805691 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805508 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 01 12:21:30.805691 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.805509 2580 manager.go:1199] Started watching for new ooms in manager Nov 01 12:21:30.806493 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.806471 2580 manager.go:319] Starting recovery of all containers Nov 01 12:21:30.817634 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.817616 2580 manager.go:324] Recovery completed Nov 01 12:21:30.822893 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.822877 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.824916 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.824898 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.824991 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.824929 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.824991 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.824941 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:30.825470 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.825460 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.825521 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.825462 2580 cpu_manager.go:226] "Starting" policy="none" Nov 01 12:21:30.825521 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.825491 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 12:21:30.825521 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.825506 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 12:21:30.826114 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.826099 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.826165 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.826132 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.826165 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.826146 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:30.827679 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.827665 2580 policy_none.go:50] "Start" Nov 01 12:21:30.827744 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.827683 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 12:21:30.827744 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.827696 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 12:21:30.829788 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.829777 2580 policy_none.go:44] "Start" Nov 01 12:21:30.857922 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.857886 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 12:21:30.862225 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862209 2580 manager.go:341] "Starting Device Plugin manager" Nov 01 12:21:30.862327 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.862244 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 12:21:30.862327 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862257 2580 server.go:87] "Starting device plugin registration server" Nov 01 12:21:30.862521 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862504 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 12:21:30.862778 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862521 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 12:21:30.862778 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862604 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 12:21:30.862778 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862678 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 12:21:30.862778 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.862686 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 12:21:30.863649 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.863630 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 12:21:30.863732 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.863676 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" Nov 01 12:21:30.886586 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.881213 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 12:21:30.886586 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.882381 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 12:21:30.886586 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.882400 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 12:21:30.886586 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.882420 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 12:21:30.886586 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.882458 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 12:21:30.924639 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.924616 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-174.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 12:21:30.924730 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.924616 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-174.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 12:21:30.925970 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.924620 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-174.us-west-2.compute.internal.18e3eb354da45c56 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:30.792000598 +0000 UTC m=+0.409325933,LastTimestamp:2026-11-01 12:21:30.792000598 +0000 UTC m=+0.409325933,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-174.us-west-2.compute.internal,}" Nov 01 12:21:30.963470 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.963401 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.964611 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.964596 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.964673 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.964624 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.964673 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.964639 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:30.964673 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.964662 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:30.973624 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.973594 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 12:21:30.973867 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.973840 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 12:21:30.973936 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:30.973827 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-174.us-west-2.compute.internal.18e3eb354f9a946f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-174.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:30.824914031 +0000 UTC m=+0.442239368,LastTimestamp:2026-11-01 12:21:30.824914031 +0000 UTC m=+0.442239368,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-174.us-west-2.compute.internal,}" Nov 01 12:21:30.984012 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.983992 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal"] Nov 01 12:21:30.984078 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.984048 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.984895 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.984881 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.984960 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.984909 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.984960 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.984918 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:30.986476 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.986461 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 12:21:30.986534 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.986498 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcbjm" Nov 01 12:21:30.987868 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.987856 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.988013 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.987999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:30.988061 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.988025 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.989262 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.989247 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.989361 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.989267 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.989361 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.989277 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:30.989361 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.989314 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.989361 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.989335 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.989361 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.989345 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:30.991432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.991417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:30.991521 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.991440 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:21:30.992170 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.992154 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:21:30.992245 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.992180 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:21:30.992245 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:30.992189 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:21:31.017879 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:31.017864 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.020162 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:31.020147 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.071582 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.071563 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:21:31.073103 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.073089 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:21:31.077661 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.077649 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.077705 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:31.077668 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-174.us-west-2.compute.internal\": node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" Nov 01 12:21:31.107590 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.107569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.107661 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.107595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c12124bf0630fd70bf94c8ced58f02e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"2c12124bf0630fd70bf94c8ced58f02e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.107661 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.107612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.205534 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.205516 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.208705 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.208691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.208753 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.208714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.208753 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.208730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c12124bf0630fd70bf94c8ced58f02e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"2c12124bf0630fd70bf94c8ced58f02e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.208823 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.208790 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.208855 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.208790 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c12124bf0630fd70bf94c8ced58f02e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"2c12124bf0630fd70bf94c8ced58f02e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.208855 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.208794 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.266271 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.266225 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 12:21:31.267331 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.267319 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.320222 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.320203 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 12:21:31.323331 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.323310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.325467 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.325454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:21:31.727721 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.727694 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 12:21:31.728156 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.727798 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 12:21:31.728156 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:31.727866 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://abc0a0f33396342028b8d407fe49f7f2-4917c070ca97fa5c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-174.us-west-2.compute.internal.18e3eb354f9b0c1e\": read tcp 10.0.140.174:45238->3.131.42.116:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-174.us-west-2.compute.internal.18e3eb354f9b0c1e default 5062 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-174.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:30 +0000 UTC,LastTimestamp:2026-11-01 12:21:30.984921485 +0000 UTC m=+0.602246820,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-174.us-west-2.compute.internal,}" Nov 01 12:21:31.776088 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.776064 2580 apiserver.go:52] "Watching apiserver" Nov 01 12:21:31.804699 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.804679 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 12:21:31.992762 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.992682 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 12:16:31 +0000 UTC" deadline="2028-06-03 09:57:29.973010985 +0000 UTC" Nov 01 12:21:31.992762 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:31.992722 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13917h35m57.980291657s" Nov 01 12:21:32.033643 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.033627 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 12:21:32.033919 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.033902 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-k6w7r","openshift-multus/multus-v4qqn","openshift-network-diagnostics/network-check-target-56m2d","kube-system/konnectivity-agent-6kq7t","kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-t5ghs","openshift-cluster-node-tuning-operator/tuned-7s548","openshift-image-registry/node-ca-kc9l9","openshift-multus/multus-additional-cni-plugins-2dcr2","openshift-ovn-kubernetes/ovnkube-node-jn9jc","kube-system/global-pull-secret-syncer-x8prg","openshift-network-operator/iptables-alerter-w6d8r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss"] Nov 01 12:21:32.036562 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.036547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.038827 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.038806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.041011 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.040992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:32.041077 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.041055 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:32.043273 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.043260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.045447 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.045430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.045539 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.045483 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:32.047718 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.047693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.049990 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.049973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.052326 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.052301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.054946 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.054924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.057676 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.057658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.057758 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.057712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:32.060102 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.060083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.063093 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.063072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.088855 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.088828 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.088954 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.088899 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.088954 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.088937 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfjvd\"" Nov 01 12:21:32.090600 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.090571 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 12:21:32.090759 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.090741 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z9pcl\"" Nov 01 12:21:32.090855 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.090799 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.090897 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.090871 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.090931 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.090902 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 12:21:32.094716 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.094698 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 12:21:32.094923 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.094908 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7twmh\"" Nov 01 12:21:32.095011 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.094998 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 12:21:32.099673 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.099657 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9cl4q\"" Nov 01 12:21:32.099753 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.099711 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.106106 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.106088 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 12:21:32.114392 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.114478 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:32.114478 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-lib-modules\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.114580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.114580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-daemon-config\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.114580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-etc-kubernetes\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.114716 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.114716 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-host-slash\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.114716 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btj8\" (UniqueName: \"kubernetes.io/projected/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-kube-api-access-5btj8\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.114716 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-conf\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.114716 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/678c9f2a-d55a-4f62-913c-650cbf020460-serviceca\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8h6\" (UniqueName: \"kubernetes.io/projected/678c9f2a-d55a-4f62-913c-650cbf020460-kube-api-access-zb8h6\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cnibin\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-systemd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-netd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-env-overrides\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-tmp\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.114876 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-config\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cnibin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkcpc\" (UniqueName: \"kubernetes.io/projected/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-kube-api-access-vkcpc\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-kubelet\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-socket-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.114976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmvr\" (UniqueName: \"kubernetes.io/projected/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-kube-api-access-zzmvr\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-slash\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-log-socket\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-multus\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-hostroot\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysconfig\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-systemd\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.115116 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-systemd-units\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbt5\" (UniqueName: \"kubernetes.io/projected/fb547296-0fb6-4120-98ad-31e2f082eaca-kube-api-access-vjbt5\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-modprobe-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-etc-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-node-log\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-dbus\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-system-cni-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-sys-fs\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb547296-0fb6-4120-98ad-31e2f082eaca-hosts-file\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb547296-0fb6-4120-98ad-31e2f082eaca-tmp-dir\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-kubelet\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-ovn\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-bin\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cni-binary-copy\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-socket-dir-parent\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-netns\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-kubernetes\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.115564 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85crl\" (UniqueName: \"kubernetes.io/projected/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-kube-api-access-85crl\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9170b98-0937-44e0-86d9-73821df95fef-konnectivity-ca\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-conf-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prg5z\" (UniqueName: \"kubernetes.io/projected/c1e8956c-23c1-4d94-8574-62fb54e26e9f-kube-api-access-prg5z\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-iptables-alerter-script\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-sys\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-host\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ch78\" (UniqueName: \"kubernetes.io/projected/ee10a231-11cb-42d1-971f-08ebea7d040c-kube-api-access-5ch78\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9170b98-0937-44e0-86d9-73821df95fef-agent-certs\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678c9f2a-d55a-4f62-913c-650cbf020460-host\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovn-node-metrics-cert\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.116045 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-var-lib-kubelet\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-netns\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.115989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-script-lib\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-system-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-kubelet-config\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-device-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-etc-tuned\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-multus-certs\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-var-lib-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-bin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-registration-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-run\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc782\" (UniqueName: \"kubernetes.io/projected/4b5b85d1-29a8-459e-86df-38b27c994538-kube-api-access-fc782\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.116530 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.116311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-k8s-cni-cncf-io\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.137149 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.137132 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 12:21:32.137234 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.137168 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m69v5" Nov 01 12:21:32.198085 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198065 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 12:21:32.198253 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198236 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.198820 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198794 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 12:21:32.198921 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198824 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 12:21:32.198921 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198830 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 12:21:32.198921 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198865 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.198921 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198915 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198924 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198958 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198963 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198970 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qrhfv\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198992 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.198992 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.199035 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.199120 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.199063 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.199956 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.199930 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 12:21:32.200036 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.199955 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 12:21:32.200036 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.200022 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 12:21:32.200133 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.200050 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsq77\"" Nov 01 12:21:32.200133 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.199939 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhf7p\"" Nov 01 12:21:32.200133 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.200111 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhblh\"" Nov 01 12:21:32.200492 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.200473 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 12:21:32.200565 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.200541 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2z6g\"" Nov 01 12:21:32.216481 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-sys\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.216576 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-host\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.216576 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216576 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216576 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ch78\" (UniqueName: \"kubernetes.io/projected/ee10a231-11cb-42d1-971f-08ebea7d040c-kube-api-access-5ch78\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.216576 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9170b98-0937-44e0-86d9-73821df95fef-agent-certs\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216591 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-host\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216603 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216628 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-sys\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678c9f2a-d55a-4f62-913c-650cbf020460-host\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216661 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678c9f2a-d55a-4f62-913c-650cbf020460-host\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216628 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovn-node-metrics-cert\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-var-lib-kubelet\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-netns\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-netns\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216788 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.216817 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216795 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-var-lib-kubelet\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-script-lib\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-system-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216872 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216954 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-system-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.216968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-kubelet-config\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-device-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-etc-tuned\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217064 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-device-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-multus-certs\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217116 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217134 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-multus-certs\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217088 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-kubelet-config\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-var-lib-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-bin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-var-lib-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.217432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-registration-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217231 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-bin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217253 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-run\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc782\" (UniqueName: \"kubernetes.io/projected/4b5b85d1-29a8-459e-86df-38b27c994538-kube-api-access-fc782\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-k8s-cni-cncf-io\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217357 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-run\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217307 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-registration-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217371 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-lib-modules\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.217453 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217456 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-script-lib\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-daemon-config\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-etc-kubernetes\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218142 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217532 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-k8s-cni-cncf-io\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.217561 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:32.71753992 +0000 UTC m=+2.334865260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217577 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-etc-kubernetes\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-host-slash\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btj8\" (UniqueName: \"kubernetes.io/projected/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-kube-api-access-5btj8\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217649 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-lib-modules\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.217652 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.217719 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret podName:f93d1106-ec10-4183-b91c-bdbf0ea5fa9a nodeName:}" failed. No retries permitted until 2026-11-01 12:21:32.717703956 +0000 UTC m=+2.335029287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret") pod "global-pull-secret-syncer-x8prg" (UID: "f93d1106-ec10-4183-b91c-bdbf0ea5fa9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-conf\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/678c9f2a-d55a-4f62-913c-650cbf020460-serviceca\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-conf\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8h6\" (UniqueName: \"kubernetes.io/projected/678c9f2a-d55a-4f62-913c-650cbf020460-kube-api-access-zb8h6\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cnibin\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217847 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cnibin\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.218975 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217881 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-host-slash\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-systemd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-netd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.217958 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-netd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218102 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-systemd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-env-overrides\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-tmp\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-config\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-daemon-config\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cnibin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcpc\" (UniqueName: \"kubernetes.io/projected/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-kube-api-access-vkcpc\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-kubelet\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-socket-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmvr\" (UniqueName: \"kubernetes.io/projected/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-kube-api-access-zzmvr\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218311 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/678c9f2a-d55a-4f62-913c-650cbf020460-serviceca\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-slash\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-log-socket\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.219838 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-multus\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-hostroot\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysconfig\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218505 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218567 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-log-socket\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218571 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-kubelet\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218606 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cnibin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218629 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-slash\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218653 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-hostroot\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-multus\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218682 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218735 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysconfig\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-systemd\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-systemd\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-systemd-units\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbt5\" (UniqueName: \"kubernetes.io/projected/fb547296-0fb6-4120-98ad-31e2f082eaca-kube-api-access-vjbt5\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-modprobe-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.220761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-etc-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-node-log\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-dbus\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-system-cni-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218895 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-systemd-units\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218857 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-config\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-sys-fs\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb547296-0fb6-4120-98ad-31e2f082eaca-hosts-file\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219224 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-node-log\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-env-overrides\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb547296-0fb6-4120-98ad-31e2f082eaca-tmp-dir\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219261 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-system-cni-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-kubelet\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219295 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-etc-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-ovn\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-bin\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-dbus\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cni-binary-copy\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-socket-dir-parent\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-netns\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-kubernetes\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219591 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb547296-0fb6-4120-98ad-31e2f082eaca-tmp-dir\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85crl\" (UniqueName: \"kubernetes.io/projected/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-kube-api-access-85crl\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.218920 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-socket-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9170b98-0937-44e0-86d9-73821df95fef-konnectivity-ca\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219686 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-kubernetes\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-conf-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219740 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-socket-dir-parent\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prg5z\" (UniqueName: \"kubernetes.io/projected/c1e8956c-23c1-4d94-8574-62fb54e26e9f-kube-api-access-prg5z\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-iptables-alerter-script\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-kubelet\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219153 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-modprobe-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219779 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-netns\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.221981 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219865 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-sys-fs\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219865 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cni-binary-copy\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219914 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb547296-0fb6-4120-98ad-31e2f082eaca-hosts-file\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219936 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-ovn\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.219960 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-bin\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.220355 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-conf-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.220463 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.220518 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.221672 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-tmp\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.221753 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovn-node-metrics-cert\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.221846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9170b98-0937-44e0-86d9-73821df95fef-agent-certs\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.222479 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9170b98-0937-44e0-86d9-73821df95fef-konnectivity-ca\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.222522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.222477 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-etc-tuned\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.222868 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.222575 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-iptables-alerter-script\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.272281 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.272217 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 12:21:32.272281 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.272239 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 12:21:32.272281 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.272254 2580 projected.go:196] Error preparing data for projected volume kube-api-access-drpvd for pod openshift-network-diagnostics/network-check-target-56m2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:32.272507 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.272310 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd podName:52417111-186c-4db8-9b3b-3b0315901dc2 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:32.772292633 +0000 UTC m=+2.389617983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-drpvd" (UniqueName: "kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd") pod "network-check-target-56m2d" (UID: "52417111-186c-4db8-9b3b-3b0315901dc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:32.274955 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.274523 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ch78\" (UniqueName: \"kubernetes.io/projected/ee10a231-11cb-42d1-971f-08ebea7d040c-kube-api-access-5ch78\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.274955 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.274718 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc782\" (UniqueName: \"kubernetes.io/projected/4b5b85d1-29a8-459e-86df-38b27c994538-kube-api-access-fc782\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.274955 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.274846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8h6\" (UniqueName: \"kubernetes.io/projected/678c9f2a-d55a-4f62-913c-650cbf020460-kube-api-access-zb8h6\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.274955 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.274920 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btj8\" (UniqueName: \"kubernetes.io/projected/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-kube-api-access-5btj8\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.275455 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.275436 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmvr\" (UniqueName: \"kubernetes.io/projected/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-kube-api-access-zzmvr\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.276089 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.276063 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcpc\" (UniqueName: \"kubernetes.io/projected/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-kube-api-access-vkcpc\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.276725 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.276701 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prg5z\" (UniqueName: \"kubernetes.io/projected/c1e8956c-23c1-4d94-8574-62fb54e26e9f-kube-api-access-prg5z\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.276891 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.276873 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85crl\" (UniqueName: \"kubernetes.io/projected/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-kube-api-access-85crl\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.277021 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.277002 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbt5\" (UniqueName: \"kubernetes.io/projected/fb547296-0fb6-4120-98ad-31e2f082eaca-kube-api-access-vjbt5\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.349909 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.349889 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:21:32.366631 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.366612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:21:32.383567 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.383539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v4qqn" Nov 01 12:21:32.400582 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.400560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:32.416457 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.416437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:21:32.427455 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.427436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:21:32.445622 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.445603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:21:32.470438 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.470415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:32.472522 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.472505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:21:32.481541 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.481524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:21:32.515712 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:32.515684 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c12124bf0630fd70bf94c8ced58f02e.slice/crio-f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6 WatchSource:0}: Error finding container f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6: Status 404 returned error can't find the container with id f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6 Nov 01 12:21:32.516452 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:32.516410 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee10a231_11cb_42d1_971f_08ebea7d040c.slice/crio-a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537 WatchSource:0}: Error finding container a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537: Status 404 returned error can't find the container with id a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537 Nov 01 12:21:32.523577 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.523562 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:21:32.548359 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:32.548337 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b76893291fa1dd017b75cc7b5ec1b7d.slice/crio-bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7 WatchSource:0}: Error finding container bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7: Status 404 returned error can't find the container with id bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7 Nov 01 12:21:32.607166 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.607144 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:21:32.723693 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.723670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:32.723803 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.723711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:32.723803 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.723794 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:32.723874 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.723813 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:32.723874 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.723839 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret podName:f93d1106-ec10-4183-b91c-bdbf0ea5fa9a nodeName:}" failed. No retries permitted until 2026-11-01 12:21:33.723825299 +0000 UTC m=+3.341150628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret") pod "global-pull-secret-syncer-x8prg" (UID: "f93d1106-ec10-4183-b91c-bdbf0ea5fa9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:32.723874 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.723864 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:33.723851093 +0000 UTC m=+3.341176416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:32.824451 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.824378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:32.824892 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.824519 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 12:21:32.824892 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.824534 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 12:21:32.824892 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.824543 2580 projected.go:196] Error preparing data for projected volume kube-api-access-drpvd for pod openshift-network-diagnostics/network-check-target-56m2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:32.824892 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:32.824597 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd podName:52417111-186c-4db8-9b3b-3b0315901dc2 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:33.824585761 +0000 UTC m=+3.441911085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-drpvd" (UniqueName: "kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd") pod "network-check-target-56m2d" (UID: "52417111-186c-4db8-9b3b-3b0315901dc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:32.888234 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.888185 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerStarted","Data":"bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7"} Nov 01 12:21:32.889286 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.889261 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"2c12124bf0630fd70bf94c8ced58f02e","Type":"ContainerStarted","Data":"f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6"} Nov 01 12:21:32.890325 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:32.890303 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" event={"ID":"ee10a231-11cb-42d1-971f-08ebea7d040c","Type":"ContainerStarted","Data":"a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537"} Nov 01 12:21:33.140590 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.140498 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:32 +0000 UTC" deadline="2028-06-20 05:53:32.101685923 +0000 UTC" Nov 01 12:21:33.140590 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.140540 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14321h31m58.961148079s" Nov 01 12:21:33.194884 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.194858 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:21:33.306701 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.306676 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb547296_0fb6_4120_98ad_31e2f082eaca.slice/crio-ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf WatchSource:0}: Error finding container ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf: Status 404 returned error can't find the container with id ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf Nov 01 12:21:33.360806 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.360779 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e7bd03_73c7_4c34_b0d6_102d25d269b4.slice/crio-4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483 WatchSource:0}: Error finding container 4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483: Status 404 returned error can't find the container with id 4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483 Nov 01 12:21:33.362405 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.362378 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5b85d1_29a8_459e_86df_38b27c994538.slice/crio-96ac46484ef2fb536c48a2423f03766f9bb108122fc71707b3184a1bb8a2ab5a WatchSource:0}: Error finding container 96ac46484ef2fb536c48a2423f03766f9bb108122fc71707b3184a1bb8a2ab5a: Status 404 returned error can't find the container with id 96ac46484ef2fb536c48a2423f03766f9bb108122fc71707b3184a1bb8a2ab5a Nov 01 12:21:33.363358 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.363245 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e8956c_23c1_4d94_8574_62fb54e26e9f.slice/crio-743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691 WatchSource:0}: Error finding container 743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691: Status 404 returned error can't find the container with id 743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691 Nov 01 12:21:33.407998 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.407972 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678c9f2a_d55a_4f62_913c_650cbf020460.slice/crio-b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0 WatchSource:0}: Error finding container b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0: Status 404 returned error can't find the container with id b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0 Nov 01 12:21:33.428801 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.428779 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a919c6_24a0_40a2_96b9_ea4e7c7098a6.slice/crio-64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd WatchSource:0}: Error finding container 64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd: Status 404 returned error can't find the container with id 64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd Nov 01 12:21:33.451110 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.451090 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3a8cd4_87b7_4dce_9109_6a3e40960f4c.slice/crio-81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398 WatchSource:0}: Error finding container 81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398: Status 404 returned error can't find the container with id 81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398 Nov 01 12:21:33.478305 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:21:33.478284 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9170b98_0937_44e0_86d9_73821df95fef.slice/crio-fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c WatchSource:0}: Error finding container fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c: Status 404 returned error can't find the container with id fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c Nov 01 12:21:33.730753 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.730680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:33.730753 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.730742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:33.730958 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.730854 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:33.730958 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.730895 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:33.730958 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.730927 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:35.730907874 +0000 UTC m=+5.348233218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:33.730958 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.730947 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret podName:f93d1106-ec10-4183-b91c-bdbf0ea5fa9a nodeName:}" failed. No retries permitted until 2026-11-01 12:21:35.730936801 +0000 UTC m=+5.348262125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret") pod "global-pull-secret-syncer-x8prg" (UID: "f93d1106-ec10-4183-b91c-bdbf0ea5fa9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:33.831988 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.831948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:33.832631 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.832193 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 12:21:33.832631 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.832216 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 12:21:33.832631 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.832231 2580 projected.go:196] Error preparing data for projected volume kube-api-access-drpvd for pod openshift-network-diagnostics/network-check-target-56m2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:33.832631 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.832337 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd podName:52417111-186c-4db8-9b3b-3b0315901dc2 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:35.83229959 +0000 UTC m=+5.449624929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-drpvd" (UniqueName: "kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd") pod "network-check-target-56m2d" (UID: "52417111-186c-4db8-9b3b-3b0315901dc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:33.882774 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.882738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:33.882917 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.882889 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:33.883021 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.883006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:33.883139 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.883111 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:33.883213 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.883199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:33.883296 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:33.883279 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:33.893948 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.893914 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerStarted","Data":"4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483"} Nov 01 12:21:33.895441 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.895409 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6kq7t" event={"ID":"a9170b98-0937-44e0-86d9-73821df95fef","Type":"ContainerStarted","Data":"fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c"} Nov 01 12:21:33.897545 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.897520 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398"} Nov 01 12:21:33.900629 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.900598 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v4qqn" event={"ID":"c1e8956c-23c1-4d94-8574-62fb54e26e9f","Type":"ContainerStarted","Data":"743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691"} Nov 01 12:21:33.902528 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.902503 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7s548" event={"ID":"4b5b85d1-29a8-459e-86df-38b27c994538","Type":"ContainerStarted","Data":"96ac46484ef2fb536c48a2423f03766f9bb108122fc71707b3184a1bb8a2ab5a"} Nov 01 12:21:33.904005 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.903981 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k6w7r" event={"ID":"fb547296-0fb6-4120-98ad-31e2f082eaca","Type":"ContainerStarted","Data":"ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf"} Nov 01 12:21:33.905509 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.905485 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w6d8r" event={"ID":"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6","Type":"ContainerStarted","Data":"64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd"} Nov 01 12:21:33.906909 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:33.906887 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kc9l9" event={"ID":"678c9f2a-d55a-4f62-913c-650cbf020460","Type":"ContainerStarted","Data":"b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0"} Nov 01 12:21:34.140809 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:34.140724 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:32 +0000 UTC" deadline="2028-07-10 22:17:35.575900712 +0000 UTC" Nov 01 12:21:34.140809 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:34.140762 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14817h56m1.435143737s" Nov 01 12:21:35.753994 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:35.753877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:35.753994 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:35.753941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:35.754704 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.754094 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:35.754704 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.754156 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret podName:f93d1106-ec10-4183-b91c-bdbf0ea5fa9a nodeName:}" failed. No retries permitted until 2026-11-01 12:21:39.754138245 +0000 UTC m=+9.371463575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret") pod "global-pull-secret-syncer-x8prg" (UID: "f93d1106-ec10-4183-b91c-bdbf0ea5fa9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:35.754704 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.754223 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:35.754704 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.754257 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:39.7542453 +0000 UTC m=+9.371570630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:35.855308 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:35.854362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:35.855308 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.854576 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 12:21:35.855308 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.854598 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 12:21:35.855308 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.854611 2580 projected.go:196] Error preparing data for projected volume kube-api-access-drpvd for pod openshift-network-diagnostics/network-check-target-56m2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:35.855308 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.854671 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd podName:52417111-186c-4db8-9b3b-3b0315901dc2 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:39.854652154 +0000 UTC m=+9.471977493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-drpvd" (UniqueName: "kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd") pod "network-check-target-56m2d" (UID: "52417111-186c-4db8-9b3b-3b0315901dc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:35.883485 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:35.883004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:35.883485 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.883133 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:35.883485 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:35.883205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:35.883485 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.883287 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:35.883485 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:35.883368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:35.883485 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:35.883441 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:36.929592 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:36.929554 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerStarted","Data":"2bd02ee4d37a9714dc4e12595c7ca710b3783475c9b66478adf04a5a96b39c0c"} Nov 01 12:21:37.883333 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.883254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:37.883522 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:37.883420 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:37.883911 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.883847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:37.884028 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:37.883963 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:37.884099 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.884036 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:37.884155 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:37.884119 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:37.938719 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.938441 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b76893291fa1dd017b75cc7b5ec1b7d" containerID="2bd02ee4d37a9714dc4e12595c7ca710b3783475c9b66478adf04a5a96b39c0c" exitCode=0 Nov 01 12:21:37.938719 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.938568 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerDied","Data":"2bd02ee4d37a9714dc4e12595c7ca710b3783475c9b66478adf04a5a96b39c0c"} Nov 01 12:21:37.941765 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.941734 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"2c12124bf0630fd70bf94c8ced58f02e","Type":"ContainerStarted","Data":"bdc8ba644cbf02f75ebf41846d04eb3257fc8b693646022b94a68cfed33829f8"} Nov 01 12:21:37.945877 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.945848 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="d5e66bb48c0f508aa278506e53cd7b58a470469b9cf35cef655779acc3c25c3c" exitCode=0 Nov 01 12:21:37.945959 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:37.945889 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerDied","Data":"d5e66bb48c0f508aa278506e53cd7b58a470469b9cf35cef655779acc3c25c3c"} Nov 01 12:21:38.957054 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:38.956988 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerStarted","Data":"711c66617c821acb9f5ff2708bb2787ef058af412a699f514add92166c211f5c"} Nov 01 12:21:38.959801 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:38.959774 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" event={"ID":"ee10a231-11cb-42d1-971f-08ebea7d040c","Type":"ContainerStarted","Data":"1b580209e06099120a82fadb199735e3debbb75c0b01d68d2f58f71740cd34b1"} Nov 01 12:21:38.966816 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:38.966787 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6kq7t" event={"ID":"a9170b98-0937-44e0-86d9-73821df95fef","Type":"ContainerStarted","Data":"bc209acca2f4570a610c641cbd9b9e78340ed074c1cf681b644f3ff813477f11"} Nov 01 12:21:39.069294 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.069170 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" podStartSLOduration=8.069151535 podStartE2EDuration="8.069151535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:38.281363142 +0000 UTC m=+7.898688469" watchObservedRunningTime="2026-11-01 12:21:39.069151535 +0000 UTC m=+8.686476880" Nov 01 12:21:39.069294 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.069263 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" podStartSLOduration=8.06925493 podStartE2EDuration="8.06925493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:39.068266992 +0000 UTC m=+8.685592337" watchObservedRunningTime="2026-11-01 12:21:39.06925493 +0000 UTC m=+8.686580275" Nov 01 12:21:39.786785 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.786703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:39.786935 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.786800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:39.787007 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.786934 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:39.787062 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.787008 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:47.786988537 +0000 UTC m=+17.404313865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:39.787456 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.787433 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:39.787581 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.787491 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret podName:f93d1106-ec10-4183-b91c-bdbf0ea5fa9a nodeName:}" failed. No retries permitted until 2026-11-01 12:21:47.787475583 +0000 UTC m=+17.404800921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret") pod "global-pull-secret-syncer-x8prg" (UID: "f93d1106-ec10-4183-b91c-bdbf0ea5fa9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:39.882722 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.882691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:39.882898 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.882834 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:39.882976 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.882933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:39.883036 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.883001 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:39.883036 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.883025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:39.883167 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.883139 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:39.887632 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.887603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:39.887781 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.887766 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 12:21:39.887848 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.887788 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 12:21:39.887848 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.887800 2580 projected.go:196] Error preparing data for projected volume kube-api-access-drpvd for pod openshift-network-diagnostics/network-check-target-56m2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:39.887918 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:39.887865 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd podName:52417111-186c-4db8-9b3b-3b0315901dc2 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:47.88784685 +0000 UTC m=+17.505172190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-drpvd" (UniqueName: "kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd") pod "network-check-target-56m2d" (UID: "52417111-186c-4db8-9b3b-3b0315901dc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:39.976390 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:39.976356 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kc9l9" event={"ID":"678c9f2a-d55a-4f62-913c-650cbf020460","Type":"ContainerStarted","Data":"7bf91d1ef2807862632c4e0dad968a3766d55ed3bb4490b28ad7560434356f64"} Nov 01 12:21:40.081453 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:40.081343 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6kq7t" podStartSLOduration=4.004238718 podStartE2EDuration="9.081322798s" totalImagesPullingTime="5.07708408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:39.174294534 +0000 UTC m=+8.791619880" watchObservedRunningTime="2026-11-01 12:21:40.081322798 +0000 UTC m=+9.698648143" Nov 01 12:21:41.182450 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.182418 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 12:21:41.875622 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.875519 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T12:21:41.182446139Z","UUID":"d7760695-f129-44bc-a31e-e77fb10c4109","Handler":null,"Name":"","Endpoint":""} Nov 01 12:21:41.877672 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.877160 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 12:21:41.877672 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.877190 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 12:21:41.883920 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.883707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:41.883920 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.883851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:41.883920 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:41.883858 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:41.883920 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.883877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:41.884148 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:41.883954 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:41.884148 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:41.884027 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:41.982333 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.982275 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" event={"ID":"ee10a231-11cb-42d1-971f-08ebea7d040c","Type":"ContainerStarted","Data":"f16621e03e640a8da67218d4f27240b92ba1c32089431489d3807fb7d4b454d5"} Nov 01 12:21:41.985092 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:41.984383 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v4qqn" event={"ID":"c1e8956c-23c1-4d94-8574-62fb54e26e9f","Type":"ContainerStarted","Data":"92363786a9f15080f2fb5cf77de696bd276201d2b58136cb3c068689bd8d11c3"} Nov 01 12:21:42.093107 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.093045 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kc9l9" podStartSLOduration=5.162308913 podStartE2EDuration="11.093025316s" totalImagesPullingTime="5.930716403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:40.081714652 +0000 UTC m=+9.699039998" watchObservedRunningTime="2026-11-01 12:21:42.093025316 +0000 UTC m=+11.710350660" Nov 01 12:21:42.093907 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.093694 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-v4qqn" podStartSLOduration=2.6038022549999997 podStartE2EDuration="11.093683816s" totalImagesPullingTime="8.489881561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:42.092689564 +0000 UTC m=+11.710014909" watchObservedRunningTime="2026-11-01 12:21:42.093683816 +0000 UTC m=+11.711009161" Nov 01 12:21:42.510491 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.510362 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:42.511438 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.511415 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:42.987683 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.987648 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k6w7r" event={"ID":"fb547296-0fb6-4120-98ad-31e2f082eaca","Type":"ContainerStarted","Data":"0a8359387edfe8df1f996f15daa3da583eb598ad60571841bdc315daad755c9e"} Nov 01 12:21:42.989935 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.989886 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerStarted","Data":"86e4c4c4ee4c81538d9cec825dda3335be0045f666c1d54690148ee8132b4d29"} Nov 01 12:21:42.990279 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.990240 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:42.990679 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:42.990659 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:21:43.092261 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.092213 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k6w7r" podStartSLOduration=3.064968845 podStartE2EDuration="12.092199747s" totalImagesPullingTime="9.027230902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:43.091898386 +0000 UTC m=+12.709223730" watchObservedRunningTime="2026-11-01 12:21:43.092199747 +0000 UTC m=+12.709525090" Nov 01 12:21:43.883443 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.883416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:43.883763 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:43.883547 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:43.883763 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.883622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:43.883763 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:43.883686 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:43.883763 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.883734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:43.883910 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:43.883812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:43.993904 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.993865 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w6d8r" event={"ID":"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6","Type":"ContainerStarted","Data":"0f57909b16d12244ccde52df1908d6de323db6a03985a7b69dee69f48739b18d"} Nov 01 12:21:43.995937 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.995906 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="86e4c4c4ee4c81538d9cec825dda3335be0045f666c1d54690148ee8132b4d29" exitCode=0 Nov 01 12:21:43.996059 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.995998 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerDied","Data":"86e4c4c4ee4c81538d9cec825dda3335be0045f666c1d54690148ee8132b4d29"} Nov 01 12:21:43.998780 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:43.998706 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" event={"ID":"ee10a231-11cb-42d1-971f-08ebea7d040c","Type":"ContainerStarted","Data":"6b737dfdfba8bd18cbce04a2016b4d468069c53e527136b92ca355a099367f1f"} Nov 01 12:21:44.097316 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:44.097219 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w6d8r" podStartSLOduration=4.387331256 podStartE2EDuration="13.097202082s" totalImagesPullingTime="8.709870826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:44.096817819 +0000 UTC m=+13.714143162" watchObservedRunningTime="2026-11-01 12:21:44.097202082 +0000 UTC m=+13.714527431" Nov 01 12:21:44.309652 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:44.309429 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" podStartSLOduration=2.565523499 podStartE2EDuration="13.309415955s" totalImagesPullingTime="10.743892456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:44.308876516 +0000 UTC m=+13.926201859" watchObservedRunningTime="2026-11-01 12:21:44.309415955 +0000 UTC m=+13.926741298" Nov 01 12:21:45.002389 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:45.002348 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7s548" event={"ID":"4b5b85d1-29a8-459e-86df-38b27c994538","Type":"ContainerStarted","Data":"98206f56cb1d97279268079015f90ac3e3b332694e0906c567dc5ec7bb6c74a9"} Nov 01 12:21:45.108289 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:45.108236 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7s548" podStartSLOduration=3.338006775 podStartE2EDuration="14.10821744s" totalImagesPullingTime="10.770210665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:45.108097369 +0000 UTC m=+14.725422714" watchObservedRunningTime="2026-11-01 12:21:45.10821744 +0000 UTC m=+14.725542785" Nov 01 12:21:45.883596 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:45.883563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:45.883757 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:45.883612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:45.883757 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:45.883679 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:45.883757 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:45.883739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:45.883873 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:45.883838 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:45.883936 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:45.883916 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:46.006724 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:46.006676 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="51762a5bbdb98389dd94d3cd8c6c4294a875c883154173d67534c1a17f54d209" exitCode=0 Nov 01 12:21:46.007424 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:46.006781 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerDied","Data":"51762a5bbdb98389dd94d3cd8c6c4294a875c883154173d67534c1a17f54d209"} Nov 01 12:21:47.848505 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:47.848476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:47.849058 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:47.848518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:47.849058 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.848643 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:47.849058 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.848649 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:47.849058 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.848697 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret podName:f93d1106-ec10-4183-b91c-bdbf0ea5fa9a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:03.848682482 +0000 UTC m=+33.466007806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret") pod "global-pull-secret-syncer-x8prg" (UID: "f93d1106-ec10-4183-b91c-bdbf0ea5fa9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 12:21:47.849058 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.848716 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:03.848705931 +0000 UTC m=+33.466031253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 12:21:47.882823 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:47.882774 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:47.882958 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:47.882940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:47.883011 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:47.882971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:47.883011 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.882956 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:47.883107 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.883044 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:47.883165 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.883145 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:47.949344 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:47.949313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:47.949489 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.949469 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 12:21:47.949542 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.949492 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 12:21:47.949542 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.949503 2580 projected.go:196] Error preparing data for projected volume kube-api-access-drpvd for pod openshift-network-diagnostics/network-check-target-56m2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:47.949610 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:47.949560 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd podName:52417111-186c-4db8-9b3b-3b0315901dc2 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:03.949541233 +0000 UTC m=+33.566866570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-drpvd" (UniqueName: "kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd") pod "network-check-target-56m2d" (UID: "52417111-186c-4db8-9b3b-3b0315901dc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 12:21:48.013390 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:48.013363 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="782e744be06e173680177f4d007ec7d807d50d4b5655e7ee2b7664aadd3d35e8" exitCode=0 Nov 01 12:21:48.013517 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:48.013415 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerDied","Data":"782e744be06e173680177f4d007ec7d807d50d4b5655e7ee2b7664aadd3d35e8"} Nov 01 12:21:49.018061 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.017889 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"b594fe300db0fc29ef970cfbb1ff6694a35146941d7cee3fd089cd4e753af601"} Nov 01 12:21:49.018428 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.018099 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"d37ad60e7bfadf70a81db2aea362143be62888835c6fadcb22f0bd66f8fdcc57"} Nov 01 12:21:49.018428 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.018130 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"d732ae737f80eb7b3a75a3f7b7d4584281447bfcbb050a5b9e39a06313dd7d64"} Nov 01 12:21:49.018428 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.018159 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"d13d60306c45dd9988ef6b7842e665824570067d60c5656fd4e4a89a9289d1d8"} Nov 01 12:21:49.018428 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.018185 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"655fe488567b7eb647eb11b23e360b4eada04acc8650978827a8d8e7910f43ff"} Nov 01 12:21:49.018428 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.018211 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"319c9e4dcba8e6183026e9dd63354f91b1820cb3b3950b114a8a76b7a78d7571"} Nov 01 12:21:49.882868 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.882831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:49.883032 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.882877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:49.883032 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:49.882939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:49.883032 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:49.882957 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:49.883032 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:49.883022 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:49.883206 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:49.883102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:51.024200 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:51.024168 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="220fcb7356a284bc9e43dba86f136187fdec2cce2ef1df50c9ac6f6e6416ac94" exitCode=0 Nov 01 12:21:51.024726 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:51.024250 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerDied","Data":"220fcb7356a284bc9e43dba86f136187fdec2cce2ef1df50c9ac6f6e6416ac94"} Nov 01 12:21:51.027514 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:51.027496 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"e13137098261271ca2322a286b13335cf7a4211398ec0dc370df452e639ad255"} Nov 01 12:21:51.882939 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:51.882903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:51.883109 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:51.882988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:51.883109 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:51.883100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:51.883204 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:51.883111 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:51.883204 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:51.883170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:51.883275 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:51.883229 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:52.032442 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:52.032406 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="c610174ce4a5d8138365ebd8a611cc8faddaf3f71f4bc7718e8abcd02dbe2758" exitCode=0 Nov 01 12:21:52.032820 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:52.032456 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerDied","Data":"c610174ce4a5d8138365ebd8a611cc8faddaf3f71f4bc7718e8abcd02dbe2758"} Nov 01 12:21:53.037457 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.037414 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" event={"ID":"c9e7bd03-73c7-4c34-b0d6-102d25d269b4","Type":"ContainerStarted","Data":"5da7af83f91cda2bc19db271bce8112dd94e05d4008a6d3f25a3f77eced4f6d3"} Nov 01 12:21:53.040772 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.040748 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" event={"ID":"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c","Type":"ContainerStarted","Data":"8bd4a516d62969b99d773ebf0adf095acb94c11d96329f32504e3cb09499881c"} Nov 01 12:21:53.040998 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.040981 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:53.041056 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.041031 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:53.055967 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.055944 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:53.239432 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.239385 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2dcr2" podStartSLOduration=8.865662207 podStartE2EDuration="22.239370955s" totalImagesPullingTime="13.373708748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:53.192353282 +0000 UTC m=+22.809678623" watchObservedRunningTime="2026-11-01 12:21:53.239370955 +0000 UTC m=+22.856696299" Nov 01 12:21:53.350376 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.350234 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" podStartSLOduration=7.725820281 podStartE2EDuration="22.35022035s" totalImagesPullingTime="14.624400069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:53.349562057 +0000 UTC m=+22.966887428" watchObservedRunningTime="2026-11-01 12:21:53.35022035 +0000 UTC m=+22.967545693" Nov 01 12:21:53.882812 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.882781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:53.882812 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.882781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:53.883036 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:53.882902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:53.883036 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:53.882898 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:53.883036 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:53.883005 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:53.883168 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:53.883088 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:54.043239 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:54.043200 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:54.056712 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:54.056689 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:21:55.360178 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:55.360101 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5ghs"] Nov 01 12:21:55.360711 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:55.360240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:55.360711 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:55.360378 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:55.363315 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:55.363268 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56m2d"] Nov 01 12:21:55.363466 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:55.363444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:55.363587 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:55.363551 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:55.363852 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:55.363833 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8prg"] Nov 01 12:21:55.363927 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:55.363916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:55.364006 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:55.363990 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:56.882654 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:56.882619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:56.883095 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:56.882634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:56.883095 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:56.882718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:56.883095 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:56.882828 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:56.883095 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:56.882876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:56.883095 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:56.882954 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:58.883336 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:58.883228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:21:58.883336 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:58.883282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:21:58.883336 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:58.883228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:21:58.883814 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:58.883385 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t5ghs" podUID="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" Nov 01 12:21:58.883814 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:58.883438 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56m2d" podUID="52417111-186c-4db8-9b3b-3b0315901dc2" Nov 01 12:21:58.883814 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:21:58.883542 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8prg" podUID="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" Nov 01 12:21:59.435426 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:59.435393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:21:59.579791 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:59.579758 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:21:59.738512 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:21:59.738481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:00.735729 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.735707 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeReady" Nov 01 12:22:00.736070 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.735814 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 12:22:00.839893 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.839862 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tv5nb"] Nov 01 12:22:00.852771 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.852751 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l64fw"] Nov 01 12:22:00.852894 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.852877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:00.857918 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.857897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:00.881257 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.881230 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tv5nb"] Nov 01 12:22:00.881951 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.881925 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l64fw"] Nov 01 12:22:00.883636 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.883608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:00.883636 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.883631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:00.883906 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.883882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:00.901665 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.901644 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 12:22:00.901665 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.901640 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 12:22:00.901810 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.901722 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lt5kb\"" Nov 01 12:22:00.909219 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.906106 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 12:22:00.909219 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.906155 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 12:22:00.928483 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.928458 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-86gmg\"" Nov 01 12:22:00.928562 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.928536 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 12:22:00.933297 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.933280 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 12:22:00.941670 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.941647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtwh\" (UniqueName: \"kubernetes.io/projected/825516f7-6409-4911-835f-7e9abb4177b5-kube-api-access-txtwh\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:00.941756 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.941705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f171cc-8de9-408d-9e4e-b9b20247048a-config-volume\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:00.941756 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.941723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:00.941877 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.941856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qw5h\" (UniqueName: \"kubernetes.io/projected/b2f171cc-8de9-408d-9e4e-b9b20247048a-kube-api-access-9qw5h\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:00.941922 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.941891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825516f7-6409-4911-835f-7e9abb4177b5-cert\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:00.941973 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.941921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f171cc-8de9-408d-9e4e-b9b20247048a-tmp-dir\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:00.947739 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.947718 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 12:22:00.947836 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.947778 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\"" Nov 01 12:22:00.947836 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.947788 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\"" Nov 01 12:22:00.947836 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.947807 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 12:22:00.947836 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.947816 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9t2fc\"" Nov 01 12:22:00.947974 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:00.947827 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 12:22:01.043020 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.042987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw5h\" (UniqueName: \"kubernetes.io/projected/b2f171cc-8de9-408d-9e4e-b9b20247048a-kube-api-access-9qw5h\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.043020 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825516f7-6409-4911-835f-7e9abb4177b5-cert\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:01.043208 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f171cc-8de9-408d-9e4e-b9b20247048a-tmp-dir\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.043208 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtwh\" (UniqueName: \"kubernetes.io/projected/825516f7-6409-4911-835f-7e9abb4177b5-kube-api-access-txtwh\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:01.043208 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f171cc-8de9-408d-9e4e-b9b20247048a-config-volume\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.043208 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.043438 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:01.043352 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:01.043438 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:01.043419 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:01.543401696 +0000 UTC m=+31.160727021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:01.043754 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043727 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f171cc-8de9-408d-9e4e-b9b20247048a-tmp-dir\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.043934 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.043915 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f171cc-8de9-408d-9e4e-b9b20247048a-config-volume\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.047057 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.046953 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825516f7-6409-4911-835f-7e9abb4177b5-cert\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:01.098906 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.098872 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw5h\" (UniqueName: \"kubernetes.io/projected/b2f171cc-8de9-408d-9e4e-b9b20247048a-kube-api-access-9qw5h\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.099022 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.098948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtwh\" (UniqueName: \"kubernetes.io/projected/825516f7-6409-4911-835f-7e9abb4177b5-kube-api-access-txtwh\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:01.171111 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.171077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:01.541374 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.541346 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l64fw"] Nov 01 12:22:01.544790 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:22:01.544757 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825516f7_6409_4911_835f_7e9abb4177b5.slice/crio-f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897 WatchSource:0}: Error finding container f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897: Status 404 returned error can't find the container with id f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897 Nov 01 12:22:01.546717 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:01.546700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:01.546821 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:01.546810 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:01.546862 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:01.546853 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.546842144 +0000 UTC m=+32.164167479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:02.060514 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:02.060482 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l64fw" event={"ID":"825516f7-6409-4911-835f-7e9abb4177b5","Type":"ContainerStarted","Data":"f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897"} Nov 01 12:22:02.553255 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:02.553218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:02.553429 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:02.553382 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:02.553479 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:02.553441 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:04.553427338 +0000 UTC m=+34.170752663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:02.764856 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:02.764809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:02.876676 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:02.876573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:03.862708 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:03.862620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:03.862708 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:03.862673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:03.863172 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:03.862788 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 12:22:03.863172 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:03.862870 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:35.862848954 +0000 UTC m=+65.480174280 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : secret "metrics-daemon-secret" not found Nov 01 12:22:03.865644 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:03.865618 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:03.910218 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:03.910188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:03.964216 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:03.963784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:03.966952 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:03.966926 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:04.083822 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:04.083791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:04.194202 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:04.194120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:04.194934 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:04.194909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:04.273860 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:04.273829 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8prg"] Nov 01 12:22:04.279594 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:22:04.279564 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93d1106_ec10_4183_b91c_bdbf0ea5fa9a.slice/crio-e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081 WatchSource:0}: Error finding container e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081: Status 404 returned error can't find the container with id e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081 Nov 01 12:22:04.561548 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:04.561522 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56m2d"] Nov 01 12:22:04.564249 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:22:04.564224 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52417111_186c_4db8_9b3b_3b0315901dc2.slice/crio-6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e WatchSource:0}: Error finding container 6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e: Status 404 returned error can't find the container with id 6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e Nov 01 12:22:04.566732 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:04.566708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:04.566854 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:04.566839 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:04.566907 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:04.566895 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:08.566877024 +0000 UTC m=+38.184202364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:05.070859 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:05.070811 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l64fw" event={"ID":"825516f7-6409-4911-835f-7e9abb4177b5","Type":"ContainerStarted","Data":"c24d0c0668bb03d965c5c8a275350a2c52dbe122756ef15a069a604862e114c0"} Nov 01 12:22:05.074606 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:05.074577 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56m2d" event={"ID":"52417111-186c-4db8-9b3b-3b0315901dc2","Type":"ContainerStarted","Data":"6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e"} Nov 01 12:22:05.075567 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:05.075545 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8prg" event={"ID":"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a","Type":"ContainerStarted","Data":"e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081"} Nov 01 12:22:05.175956 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:05.175907 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l64fw" podStartSLOduration=2.190002272 podStartE2EDuration="5.175892334s" totalImagesPullingTime="2.985890062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:05.175162669 +0000 UTC m=+34.792488016" watchObservedRunningTime="2026-11-01 12:22:05.175892334 +0000 UTC m=+34.793217671" Nov 01 12:22:06.319453 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.319414 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5q4xn"] Nov 01 12:22:06.322615 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.322571 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.371268 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.371236 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 12:22:06.381275 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.380928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.381275 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.380973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzlz\" (UniqueName: \"kubernetes.io/projected/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-api-access-zlzlz\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.381275 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.381131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.381275 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.381164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-data-volume\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.381275 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.381192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-crio-socket\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.417633 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.417531 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 12:22:06.417633 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.417531 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 12:22:06.417633 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.417604 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-thbc5\"" Nov 01 12:22:06.417907 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.417645 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 12:22:06.418580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.418557 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5q4xn"] Nov 01 12:22:06.482506 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-data-volume\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.482659 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-crio-socket\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.482659 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.482659 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzlz\" (UniqueName: \"kubernetes.io/projected/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-api-access-zlzlz\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.482659 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.482859 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.482715 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:06.482859 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.482775 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.982756856 +0000 UTC m=+36.600082196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:06.482859 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-data-volume\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.483018 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.482714 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 12:22:06.483018 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.482903 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.982890486 +0000 UTC m=+36.600215828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : configmap "kube-rbac-proxy" not found Nov 01 12:22:06.483018 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.482912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-crio-socket\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.526664 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.526589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:06.544690 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.544638 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzlz\" (UniqueName: \"kubernetes.io/projected/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-api-access-zlzlz\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.562281 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.562253 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-r5bbx-8fck5"] Nov 01 12:22:06.565512 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.565481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.613998 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.613965 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jtrgb\"" Nov 01 12:22:06.618580 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.618555 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-r5bbx-8fck5"] Nov 01 12:22:06.618704 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.618691 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 12:22:06.677661 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.677626 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 12:22:06.683987 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.683955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.684114 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.683997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.684168 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.684112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.684168 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.684137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.784690 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.784653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.784837 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.784697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.784837 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.784736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.784837 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.784760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.785412 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.785381 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.785928 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.785906 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.786042 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.786023 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.842143 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.842080 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.879892 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.879861 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:06.985902 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.985868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.986113 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.985966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:06.986113 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.986036 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 12:22:06.986113 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.986041 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:06.986113 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.986087 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:07.986072722 +0000 UTC m=+37.603398044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : configmap "kube-rbac-proxy" not found Nov 01 12:22:06.986318 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:06.986123 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:07.98610372 +0000 UTC m=+37.603429059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:06.988113 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:06.988085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:07.137877 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.137800 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:07.257599 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.257560 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd"] Nov 01 12:22:07.261831 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.261808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:07.267160 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.267127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:07.267278 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.267258 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-r5bbx-8fck5"] Nov 01 12:22:07.271714 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:22:07.271661 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129138ad_2f7a_4994_8693_61c7b660f53f.slice/crio-9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d WatchSource:0}: Error finding container 9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d: Status 404 returned error can't find the container with id 9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d Nov 01 12:22:07.314978 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.314460 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 12:22:07.314978 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.314553 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-svx22\"" Nov 01 12:22:07.314978 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.314659 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 12:22:07.316345 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.316321 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd"] Nov 01 12:22:07.379967 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.379937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:07.388744 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.388676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69p6p\" (UniqueName: \"kubernetes.io/projected/2a422fe5-8a7c-482a-9564-56151c0ca16c-kube-api-access-69p6p\") pod \"migrator-66df5d496b-24rdd\" (UID: \"2a422fe5-8a7c-482a-9564-56151c0ca16c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:07.490121 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.490085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69p6p\" (UniqueName: \"kubernetes.io/projected/2a422fe5-8a7c-482a-9564-56151c0ca16c-kube-api-access-69p6p\") pod \"migrator-66df5d496b-24rdd\" (UID: \"2a422fe5-8a7c-482a-9564-56151c0ca16c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:07.499893 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.499857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:07.549094 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.549062 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69p6p\" (UniqueName: \"kubernetes.io/projected/2a422fe5-8a7c-482a-9564-56151c0ca16c-kube-api-access-69p6p\") pod \"migrator-66df5d496b-24rdd\" (UID: \"2a422fe5-8a7c-482a-9564-56151c0ca16c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:07.581906 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.581872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:07.619788 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.619709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:07.953792 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.953713 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd"] Nov 01 12:22:07.961980 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:22:07.961957 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a422fe5_8a7c_482a_9564_56151c0ca16c.slice/crio-8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6 WatchSource:0}: Error finding container 8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6: Status 404 returned error can't find the container with id 8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6 Nov 01 12:22:07.994925 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.994898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:07.995041 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:07.994950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:07.995041 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:07.995017 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 12:22:07.995151 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:07.995086 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:09.995066663 +0000 UTC m=+39.612391989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : configmap "kube-rbac-proxy" not found Nov 01 12:22:07.995151 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:07.995106 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:07.995273 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:07.995161 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:09.995146558 +0000 UTC m=+39.612471881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:08.085875 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.085836 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" event={"ID":"129138ad-2f7a-4994-8693-61c7b660f53f","Type":"ContainerStarted","Data":"9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d"} Nov 01 12:22:08.087274 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.087237 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" event={"ID":"2a422fe5-8a7c-482a-9564-56151c0ca16c","Type":"ContainerStarted","Data":"8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6"} Nov 01 12:22:08.242635 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.242604 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wgsgn"] Nov 01 12:22:08.245726 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.245700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.294958 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.294930 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 12:22:08.295098 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.295067 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 12:22:08.295160 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.295122 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 12:22:08.295216 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.295163 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 12:22:08.295376 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.295351 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9qzg6\"" Nov 01 12:22:08.299921 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.299900 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wgsgn"] Nov 01 12:22:08.398041 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.398012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.398513 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.398080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-key\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.398513 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.398104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5f6\" (UniqueName: \"kubernetes.io/projected/51e6e124-97cb-448a-a1a3-27ad1f617d87-kube-api-access-qs5f6\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.436128 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.436070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:08.499581 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.499476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-key\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.499581 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.499519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5f6\" (UniqueName: \"kubernetes.io/projected/51e6e124-97cb-448a-a1a3-27ad1f617d87-kube-api-access-qs5f6\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.499746 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.499626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.500427 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.500399 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.502088 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.502069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-key\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.553719 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.553693 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5f6\" (UniqueName: \"kubernetes.io/projected/51e6e124-97cb-448a-a1a3-27ad1f617d87-kube-api-access-qs5f6\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.559379 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.559354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:08.570358 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.570291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:08.601621 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.601591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:08.601779 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:08.601698 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:08.601779 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:08.601744 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:16.601731285 +0000 UTC m=+46.219056612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:08.942984 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.942906 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wgsgn"] Nov 01 12:22:08.949380 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:08.949293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:08.952073 ip-10-0-140-174 kubenswrapper[2580]: W1101 12:22:08.952044 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e6e124_97cb_448a_a1a3_27ad1f617d87.slice/crio-3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e WatchSource:0}: Error finding container 3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e: Status 404 returned error can't find the container with id 3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e Nov 01 12:22:09.092108 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:09.092069 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56m2d" event={"ID":"52417111-186c-4db8-9b3b-3b0315901dc2","Type":"ContainerStarted","Data":"c5c8660a3d9bffb07a322286e97ffed7114ba24ea8c6c148fe4a37227b290a89"} Nov 01 12:22:09.092418 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:09.092379 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:09.093761 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:09.093707 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" event={"ID":"51e6e124-97cb-448a-a1a3-27ad1f617d87","Type":"ContainerStarted","Data":"3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e"} Nov 01 12:22:09.200574 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:09.200458 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-56m2d" podStartSLOduration=34.341012919 podStartE2EDuration="38.200438616s" totalImagesPullingTime="3.859425697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:09.19927052 +0000 UTC m=+38.816595866" watchObservedRunningTime="2026-11-01 12:22:09.200438616 +0000 UTC m=+38.817764167" Nov 01 12:22:09.945769 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:09.945739 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:10.012484 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:10.012458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:10.012552 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:10.012514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:10.012678 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:10.012661 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:10.012730 ip-10-0-140-174 kubenswrapper[2580]: E1101 12:22:10.012720 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.01270241 +0000 UTC m=+43.630027746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:10.013231 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:10.013205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:10.098055 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:10.097431 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" event={"ID":"129138ad-2f7a-4994-8693-61c7b660f53f","Type":"ContainerStarted","Data":"39a4181d8cf857e017284f9573a659a99ffc06338fe49b07c71edc7fee844281"} Nov 01 12:22:10.099321 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:10.099276 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" event={"ID":"2a422fe5-8a7c-482a-9564-56151c0ca16c","Type":"ContainerStarted","Data":"441648c831d4645f4a56268f7661ff47fa3ad71f57fe5d5336c79c5da73e70c7"} Nov 01 12:22:10.102908 ip-10-0-140-174 kubenswrapper[2580]: I1101 12:22:10.102887 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8prg" event={"ID":"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a","Type":"ContainerStarted","Data":"210fb5fda6cb59816fe327f917af89e481b9bece09c845b7be68a047c910bf02"} Nov 01 12:22:10.199888 ip-10-0-140-174 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 12:22:10.206954 ip-10-0-140-174 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 12:22:10.207142 ip-10-0-140-174 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 12:22:10.207211 ip-10-0-140-174 systemd[1]: kubelet.service: Consumed 2.263s CPU time. Nov 01 12:22:10.220820 ip-10-0-140-174 systemd[1]: Starting Kubernetes Kubelet... Nov 01 12:22:10.279012 ip-10-0-140-174 kubenswrapper[4484]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 12:22:10.279012 ip-10-0-140-174 kubenswrapper[4484]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 12:22:10.279012 ip-10-0-140-174 kubenswrapper[4484]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 12:22:10.279012 ip-10-0-140-174 kubenswrapper[4484]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 12:22:10.281036 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.280960 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.281036 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281035 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.281036 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281040 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281043 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281047 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281049 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281052 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281055 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281058 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281060 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281063 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281066 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281068 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281071 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281082 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281085 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281087 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281090 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281093 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281096 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281098 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281101 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.281134 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281104 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281107 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281110 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281113 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281115 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281118 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281121 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281124 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281127 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281129 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281132 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281135 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281137 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281140 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281143 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281146 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281149 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281152 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281154 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281157 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.281655 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281160 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281162 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281165 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281167 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281170 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281173 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281175 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281178 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281180 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281183 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281186 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281188 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281191 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281194 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281197 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281199 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281202 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281205 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281209 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281211 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.282144 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281214 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281217 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281220 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281222 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281225 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281228 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281231 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281233 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281236 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281238 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281241 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281244 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281249 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281252 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281256 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281259 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281262 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281265 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281270 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.282653 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281275 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281279 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281281 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281284 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281287 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281289 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281292 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281294 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281311 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281316 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281320 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281325 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281329 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281333 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281336 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281339 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281342 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281346 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281349 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.283119 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281352 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281355 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281358 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281360 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281363 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.281366 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282080 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282087 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282090 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282092 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282095 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282098 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282101 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282104 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282106 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282110 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282112 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282115 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282117 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282120 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.283597 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282123 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282126 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282128 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282131 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282134 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282137 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282139 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282142 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282145 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282148 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282151 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282154 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282157 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282159 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282162 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282164 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282167 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282171 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282175 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.284111 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282179 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282183 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282186 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282190 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282193 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282196 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282200 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282204 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282207 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282210 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282213 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282215 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282218 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282220 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282223 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282226 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282228 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282231 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282233 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282236 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.284584 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282238 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282241 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282243 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282246 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282249 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282251 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282254 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282256 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282259 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282261 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282264 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282266 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282269 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282272 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282274 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282277 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282279 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282282 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282285 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282288 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.285075 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282291 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282294 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282312 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282317 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282321 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282325 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282329 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282333 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282335 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282338 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282340 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282343 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282345 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282348 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282351 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282354 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282356 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282359 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282362 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.285574 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282364 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282367 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282369 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282372 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282374 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282377 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282379 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282382 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282386 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282388 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282391 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282394 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282397 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282399 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282863 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282868 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282871 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282874 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282877 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.286030 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282880 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282882 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282885 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282889 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282893 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282896 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282899 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282901 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282904 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282907 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282910 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282912 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282915 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282917 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282920 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282923 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282925 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282928 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282930 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282933 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.286532 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282935 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282938 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282941 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282943 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282946 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282948 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282951 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282954 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282957 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282960 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282962 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282965 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282967 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282970 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282972 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282975 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282977 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282982 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282985 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.287087 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282987 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282990 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282993 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282995 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.282998 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283000 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283003 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283007 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283010 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283012 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283015 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283017 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283020 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283023 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283026 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283028 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283031 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283033 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283036 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283038 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.287870 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283041 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283044 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283047 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283049 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283052 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283054 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283057 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283059 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283062 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283065 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283070 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283072 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283075 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283077 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283080 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283082 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283085 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283087 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283090 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.288387 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283092 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283096 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283100 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283103 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283105 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283108 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283110 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283113 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283116 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283118 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283121 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283124 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283127 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283129 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283133 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283137 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283139 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283142 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283144 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283147 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.288840 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283149 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283155 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283158 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283252 4484 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283267 4484 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283273 4484 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283278 4484 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283282 4484 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283285 4484 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283289 4484 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283294 4484 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283309 4484 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283312 4484 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283316 4484 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283320 4484 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283323 4484 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283326 4484 flags.go:64] FLAG: --cgroup-root="" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283329 4484 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283332 4484 flags.go:64] FLAG: --client-ca-file="" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283335 4484 flags.go:64] FLAG: --cloud-provider="external" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283338 4484 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283341 4484 flags.go:64] FLAG: --cluster-domain="" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283345 4484 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 12:22:10.289428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283348 4484 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283351 4484 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283354 4484 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283358 4484 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283361 4484 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283365 4484 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283368 4484 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283372 4484 flags.go:64] FLAG: --contention-profiling="false" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283375 4484 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283378 4484 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283382 4484 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283386 4484 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283390 4484 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283393 4484 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283396 4484 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283399 4484 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283402 4484 flags.go:64] FLAG: --enable-server="true" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283405 4484 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283409 4484 flags.go:64] FLAG: --event-burst="100" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283413 4484 flags.go:64] FLAG: --event-qps="50" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283416 4484 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283419 4484 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283423 4484 flags.go:64] FLAG: --eviction-hard="" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283427 4484 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 12:22:10.290148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283430 4484 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283433 4484 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283436 4484 flags.go:64] FLAG: --eviction-soft="" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283439 4484 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283442 4484 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283446 4484 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283448 4484 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283451 4484 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283454 4484 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283457 4484 flags.go:64] FLAG: --feature-gates="" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283461 4484 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283464 4484 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283467 4484 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283471 4484 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283474 4484 flags.go:64] FLAG: --healthz-port="10248" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283477 4484 flags.go:64] FLAG: --help="false" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283481 4484 flags.go:64] FLAG: --hostname-override="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283484 4484 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283489 4484 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283492 4484 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283495 4484 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283498 4484 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283501 4484 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 12:22:10.290800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283504 4484 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283507 4484 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283510 4484 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283513 4484 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283516 4484 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283519 4484 flags.go:64] FLAG: --kube-reserved="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283528 4484 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283532 4484 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283537 4484 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283541 4484 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283544 4484 flags.go:64] FLAG: --lock-file="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283547 4484 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283550 4484 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283553 4484 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283559 4484 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283562 4484 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283565 4484 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283567 4484 flags.go:64] FLAG: --logging-format="text" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283570 4484 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283574 4484 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283577 4484 flags.go:64] FLAG: --manifest-url="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283580 4484 flags.go:64] FLAG: --manifest-url-header="" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283585 4484 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283588 4484 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283592 4484 flags.go:64] FLAG: --max-pods="110" Nov 01 12:22:10.291375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283597 4484 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283600 4484 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283605 4484 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283608 4484 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283611 4484 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283614 4484 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283618 4484 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283625 4484 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283628 4484 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283631 4484 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283634 4484 flags.go:64] FLAG: --pod-cidr="" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283637 4484 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283640 4484 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283643 4484 flags.go:64] FLAG: --pods-per-core="0" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283646 4484 flags.go:64] FLAG: --port="10250" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283649 4484 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283652 4484 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08eb0fe37f3bce0c7" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283656 4484 flags.go:64] FLAG: --qos-reserved="" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283659 4484 flags.go:64] FLAG: --read-only-port="10255" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283662 4484 flags.go:64] FLAG: --register-node="true" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283664 4484 flags.go:64] FLAG: --register-with-taints="" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283668 4484 flags.go:64] FLAG: --registry-burst="10" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283671 4484 flags.go:64] FLAG: --registry-qps="5" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283674 4484 flags.go:64] FLAG: --reserved-cpus="" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283677 4484 flags.go:64] FLAG: --reserved-memory="" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283680 4484 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 12:22:10.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283684 4484 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283687 4484 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283690 4484 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283693 4484 flags.go:64] FLAG: --runonce="false" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283696 4484 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283699 4484 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283703 4484 flags.go:64] FLAG: --seccomp-default="false" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283706 4484 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283710 4484 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283713 4484 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283716 4484 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283720 4484 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283722 4484 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283725 4484 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283728 4484 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283731 4484 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283734 4484 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283737 4484 flags.go:64] FLAG: --system-cgroups="" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283740 4484 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283745 4484 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283749 4484 flags.go:64] FLAG: --tls-cert-file="" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283751 4484 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283755 4484 flags.go:64] FLAG: --tls-min-version="" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283758 4484 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283761 4484 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 12:22:10.292942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283764 4484 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283767 4484 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283770 4484 flags.go:64] FLAG: --v="2" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283774 4484 flags.go:64] FLAG: --version="false" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283779 4484 flags.go:64] FLAG: --vmodule="" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283783 4484 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.283786 4484 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283913 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283917 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283919 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283923 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283925 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283928 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283933 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283936 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283940 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283943 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283946 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283949 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283952 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283955 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.293575 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283957 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283960 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283963 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283966 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283968 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283971 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283973 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283976 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283978 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283981 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283984 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283988 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283992 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283995 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.283998 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284001 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284003 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284006 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284008 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.294159 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284011 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284013 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284016 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284018 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284021 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284025 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284027 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284032 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284035 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284037 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284040 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284042 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284045 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284048 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284050 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284053 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284055 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284058 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284060 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284063 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.294706 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284065 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284068 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284071 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284073 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284076 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284078 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284081 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284083 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284086 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284089 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284091 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284094 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284096 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284099 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284101 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284104 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284106 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284110 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284113 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.295283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284117 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284120 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284123 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284126 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284128 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284131 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284134 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284136 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284139 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284142 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284145 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284147 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284150 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284152 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284156 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284160 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284164 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284167 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284169 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.296031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284172 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284175 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284177 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284180 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284182 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284185 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284187 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284190 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284192 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284195 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284197 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284201 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284204 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284208 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284211 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.284237 4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 12:22:10.296715 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284273 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284277 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284280 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284283 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284287 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284289 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284292 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284295 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284310 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284314 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284317 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284320 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284323 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284326 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284329 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284331 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284334 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284337 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284339 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.297106 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284342 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284345 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284347 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284350 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284353 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284355 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284358 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284361 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284365 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284368 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284370 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284374 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284377 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284380 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284383 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284385 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284388 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284390 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284393 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284396 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.297753 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284399 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284402 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284404 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284407 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284410 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284412 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284415 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284417 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284420 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284422 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284425 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284427 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284430 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284434 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284438 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284440 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284443 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284445 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284448 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.298593 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284450 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284455 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284458 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284461 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284464 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284467 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284470 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284473 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284475 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284478 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284481 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284483 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284486 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284488 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284491 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284494 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284496 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284499 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284501 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284504 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.299116 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284506 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284509 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284511 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284514 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284516 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284519 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284522 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284525 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284527 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284530 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284532 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284535 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284537 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284541 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284544 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284546 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284550 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284553 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284556 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.299643 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284558 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284561 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284564 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284567 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284569 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284572 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284574 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284577 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.284579 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.289541 4484 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.289559 4484 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.289583 4484 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.289588 4484 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289645 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289650 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289654 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289657 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289660 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289663 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.300099 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289666 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289669 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289674 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289678 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289681 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289684 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289687 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289689 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289692 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289695 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289698 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289700 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289703 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289706 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289709 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289712 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289714 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289717 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289720 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.300567 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289722 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289725 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289734 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289737 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289740 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289742 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289745 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289747 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289750 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289752 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289755 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289758 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289760 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289763 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289765 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289768 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289770 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289773 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289776 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.301069 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289778 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289781 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289784 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289787 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289791 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289795 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289798 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289801 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289804 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289807 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289809 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289812 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289814 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289817 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289819 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289822 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289824 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289827 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289830 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.301869 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289832 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289835 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289837 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289840 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289842 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289845 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289847 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289850 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289853 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289855 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289858 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289860 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289863 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289865 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289868 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289871 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289875 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289877 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289880 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289883 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.302589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289885 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289888 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289891 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289893 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289896 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289898 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289901 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289903 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289906 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289908 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289911 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289914 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289916 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289919 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289923 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289927 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289931 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289935 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289938 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289942 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.303077 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289946 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289950 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.289954 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.289991 4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290182 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290188 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290191 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290194 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290198 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290202 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290205 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290208 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290211 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290213 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290216 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290218 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.303589 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290221 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290224 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290227 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290229 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290232 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290234 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290237 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290239 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290241 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290244 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290247 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290249 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290252 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290254 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290257 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290259 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290262 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290264 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290267 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.303965 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290271 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290275 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290279 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290282 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290285 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290288 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290291 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290294 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290313 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290317 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290319 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290322 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290325 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290327 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290330 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290332 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290335 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290337 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290340 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290342 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.304437 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290345 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290347 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290350 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290352 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290355 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290358 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290360 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290363 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290365 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290368 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290370 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290373 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290375 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290378 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290381 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290383 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290386 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290388 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290391 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290394 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.304916 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290397 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290400 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290402 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290405 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290407 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290410 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290412 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290415 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290418 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290420 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290423 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290426 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290428 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290431 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290433 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290436 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290438 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290441 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290445 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.305417 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290449 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290451 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290454 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290457 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290460 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290462 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290465 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290467 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290470 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290472 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290475 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290478 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290480 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290483 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290486 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290488 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.305905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.290513 4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290547 4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290552 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290555 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290558 4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290561 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290564 4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290566 4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290569 4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290574 4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290577 4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290579 4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290582 4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290585 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290587 4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290590 4484 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290592 4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290595 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290598 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290600 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 12:22:10.306383 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290603 4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290606 4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290609 4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290611 4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290614 4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290616 4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290619 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290622 4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290624 4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290627 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290629 4484 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290634 4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290638 4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290641 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290644 4484 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290647 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290649 4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290651 4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 12:22:10.306859 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290654 4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290656 4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290659 4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290663 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290666 4484 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290668 4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290671 4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290674 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290677 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290679 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290682 4484 feature_gate.go:386] unrecognized feature gate: Example Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290684 4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290687 4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290690 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290693 4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290695 4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290698 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290700 4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290703 4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290706 4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 12:22:10.307350 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290708 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290711 4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290713 4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290716 4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290719 4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290721 4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290724 4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290727 4484 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290730 4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290732 4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290735 4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290737 4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290740 4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290742 4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290745 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290749 4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290751 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290754 4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290756 4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290759 4484 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 12:22:10.307847 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290762 4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290766 4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290770 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290773 4484 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290776 4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290779 4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290782 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290784 4484 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290787 4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290790 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290793 4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290795 4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290801 4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290804 4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290807 4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290809 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290812 4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290815 4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290818 4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290821 4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 12:22:10.308488 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290824 4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290826 4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290828 4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290831 4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290834 4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290836 4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290839 4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290842 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:10.290844 4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.290900 4484 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.291877 4484 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.291920 4484 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.292102 4484 server.go:1014] "Starting client certificate rotation" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.292744 4484 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.292767 4484 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.292833 4484 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 12:16:31 +0000 UTC" deadline="2028-06-11 14:21:28.789864162 +0000 UTC" Nov 01 12:22:10.308969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.292902 4484 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14113h59m18.496964325s" Nov 01 12:22:10.309410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.293974 4484 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 12:22:10.309410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.295669 4484 log.go:25] "Validated CRI v1 runtime API" Nov 01 12:22:10.309410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.297359 4484 log.go:25] "Validated CRI v1 image API" Nov 01 12:22:10.309410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.298146 4484 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 12:22:10.309410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.299963 4484 fs.go:135] Filesystem UUIDs: map[6f1500bb-a167-4d5b-951f-4f6b67f05cf3:/dev/nvme0n1p3 755deffa-cdf4-4566-b077-7b875954c767:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 01 12:22:10.309589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.299987 4484 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/129138ad-2f7a-4994-8693-61c7b660f53f/volumes/kubernetes.io~projected/kube-api-access-xgltg:{mountpoint:/var/lib/kubelet/pods/129138ad-2f7a-4994-8693-61c7b660f53f/volumes/kubernetes.io~projected/kube-api-access-xgltg major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a422fe5-8a7c-482a-9564-56151c0ca16c/volumes/kubernetes.io~projected/kube-api-access-69p6p:{mountpoint:/var/lib/kubelet/pods/2a422fe5-8a7c-482a-9564-56151c0ca16c/volumes/kubernetes.io~projected/kube-api-access-69p6p major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~projected/kube-api-access-fc782:{mountpoint:/var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~projected/kube-api-access-fc782 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volumes/kubernetes.io~projected/kube-api-access-vkcpc:{mountpoint:/var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volumes/kubernetes.io~projected/kube-api-access-vkcpc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51e6e124-97cb-448a-a1a3-27ad1f617d87/volumes/kubernetes.io~projected/kube-api-access-qs5f6:{mountpoint:/var/lib/kubelet/pods/51e6e124-97cb-448a-a1a3-27ad1f617d87/volumes/kubernetes.io~projected/kube-api-access-qs5f6 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51e6e124-97cb-448a-a1a3-27ad1f617d87/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/51e6e124-97cb-448a-a1a3-27ad1f617d87/volumes/kubernetes.io~secret/signing-key major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52417111-186c-4db8-9b3b-3b0315901dc2/volumes/kubernetes.io~projected/kube-api-access-drpvd:{mountpoint:/var/lib/kubelet/pods/52417111-186c-4db8-9b3b-3b0315901dc2/volumes/kubernetes.io~projected/kube-api-access-drpvd major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/678c9f2a-d55a-4f62-913c-650cbf020460/volumes/kubernetes.io~projected/kube-api-access-zb8h6:{mountpoint:/var/lib/kubelet/pods/678c9f2a-d55a-4f62-913c-650cbf020460/volumes/kubernetes.io~projected/kube-api-access-zb8h6 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/825516f7-6409-4911-835f-7e9abb4177b5/volumes/kubernetes.io~projected/kube-api-access-txtwh:{mountpoint:/var/lib/kubelet/pods/825516f7-6409-4911-835f-7e9abb4177b5/volumes/kubernetes.io~projected/kube-api-access-txtwh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/825516f7-6409-4911-835f-7e9abb4177b5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/825516f7-6409-4911-835f-7e9abb4177b5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9170b98-0937-44e0-86d9-73821df95fef/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a9170b98-0937-44e0-86d9-73821df95fef/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2f171cc-8de9-408d-9e4e-b9b20247048a/volumes/kubernetes.io~projected/kube-api-access-9qw5h:{mountpoint:/var/lib/kubelet/pods/b2f171cc-8de9-408d-9e4e-b9b20247048a/volumes/kubernetes.io~projected/kube-api-access-9qw5h major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1e8956c-23c1-4d94-8574-62fb54e26e9f/volumes/kubernetes.io~projected/kube-api-access-prg5z:{mountpoint:/var/lib/kubelet/pods/c1e8956c-23c1-4d94-8574-62fb54e26e9f/volumes/kubernetes.io~projected/kube-api-access-prg5z major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5e66e40-bbf7-4a34-848c-0f2e7842b71f/volumes/kubernetes.io~projected/kube-api-access-85crl:{mountpoint:/var/lib/kubelet/pods/c5e66e40-bbf7-4a34-848c-0f2e7842b71f/volumes/kubernetes.io~projected/kube-api-access-85crl major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9e7bd03-73c7-4c34-b0d6-102d25d269b4/volumes/kubernetes.io~projected/kube-api-access-zzmvr:{mountpoint:/var/lib/kubelet/pods/c9e7bd03-73c7-4c34-b0d6-102d25d269b4/volumes/kubernetes.io~projected/kube-api-access-zzmvr major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6/volumes/kubernetes.io~projected/kube-api-access-5btj8:{mountpoint:/var/lib/kubelet/pods/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6/volumes/kubernetes.io~projected/kube-api-access-5btj8 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee10a231-11cb-42d1-971f-08ebea7d040c/volumes/kubernetes.io~projected/kube-api-access-5ch78:{mountpoint:/var/lib/kubelet/pods/ee10a231-11cb-42d1-971f-08ebea7d040c/volumes/kubernetes.io~projected/kube-api-access-5ch78 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b/volumes/kubernetes.io~projected/kube-api-access-zlzlz:{mountpoint:/var/lib/kubelet/pods/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b/volumes/kubernetes.io~projected/kube-api-access-zlzlz major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb547296-0fb6-4120-98ad-31e2f082eaca/volumes/kubernetes.io~projected/kube-api-access-vjbt5:{mountpoint:/var/lib/kubelet/pods/fb547296-0fb6-4120-98ad-31e2f082eaca/volumes/kubernetes.io~projected/kube-api-access-vjbt5 major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c915650d8ef94b047d4ae8453fae48a32666dfcefecdbad8dd5597e9136fc84b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5ab4bc3532506fb74af11c6227612298c376875d214d8d3724cb03db583d72b2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/e047c5e17497681574a7627f2b6db6f524b707fec07df54e573b4d02902cd538/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/4e8b29bdb9121c71033d1f451ea4c92832b9cf746d7d13fdc03ec21b97bce61f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0c67dbc089214e8fa994f1dbb3c269b70a1a5bd04e1eb67f9e7824d7863a5e94/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/e3f89bf1c756829b951a34cd0c6a424d25b34ac747f0ba899b33b6fb91fbc908/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/537287ffc640b8f4faeee630dcf137b871e8a735c40b54d3aa060cf6182e9af4/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7c435914c5364009e1d6a7eaae559e814adfcd22e6c9f4a2d79e9b7b64497171/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/46e5de4b11ab97c8d5b5f2c1ad437f026ce791412082c86869ee954c6e71d269/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5126ffcc9dd502dd27d3e8856f476d3d7a44cde01b9ea7db3200d396b0862625/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8278cc6a541f4d4c37182be81536ed4003754f783d687fd06306c2c0d96fc154/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/a59c9fb8263601410beecc082b8beae1df3dc4856954397ba0703caecfd845b7/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ba8181f0056be61a0a529ac1335e47704705b075475d356fe3568078a42f610a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/bcbf201379766f1f7b193acaed69a2dca1358ae48309aac8eafebacba2ba90ea/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b00c3b379afe1e68cffc1e80943cdf3a5cdef3cfd2f189388519ab3f94aeb6e1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/dff70a1e2483230cc252564c54521615c83a8922989a37b7ffc2c87b8aa045b1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fdd3d108e6e81525869e8c05c30b5cc7994b5f21fe6e599ddf49c50a6ca5a8d4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/db5b3bfb985119c7c8b3b4fc4f04fba72114d9eb3d5ac93617e37b5d95839bd9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5c1e6d648d7681455184a0109725fd68e6f91bc20293e2bdee42f2c167b7abf4/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cd35dde6efc030985dab264d63947f7a1e700595d61399bde55dd351e1f077d5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/eafbd3547aa316936febdd953bfc0a4efaf0a876fdf96478e2e999fcc6629ca4/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/047c22cf228691af7798c85f939dcbeb33f288cf467552b4b2140ec003930706/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/c1130a529e2edb5f9d2629f7abee30769a625b1eade8b65aa6c514bc49129393/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/3cfce27b902c191a1b4a0af4d902b823f10d715fa07e2cda9d84e4af60fc85f2/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/b8fdd5078c03f26ca5c5c175d0235485d4d74e90eb04e81239bea09cf55455d2/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/02a785de3e8ee519de8ab23a09c19ae1f0f79c433d6acbefd15fbe94f33635aa/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/8a8f26a886ee08d8b5c49b5dbf914be225ba259533a797de9f6fd628473a10da/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/021e067c665a7b6a522313547a98480f74c7fd72103167f206d0063905dbdd19/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/97588caeb84cb5c0cc0002b6b144b57393858a4e621949207ef44fa9c78e4291/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/8c75476b2784ab2ade7538f6063d748bd788de91d0ce39583df7d9bf121ef9f5/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/b387159d398a96d231f300c600d1a8a888650f5db003fc8d0f3700c8e9a98b40/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/18168075255e56ea6aa2f8bf5d4f7691254c4fa2f3419a2c99b928c04d866b0a/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/c2c19017ec851e2011b808d4737d04bd8404cd3a0ecd63c5ee964739a6b0355d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/4c664eae1baf5172935bd46f02529d434bc4c7b42a22341548e8672a1e6c40a5/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1b2ef995da2050bfa189ebbf9d4ff90d04e329455c506b1e1e0224f1ec43edc1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/134a74d0a6f61717dc5e0939754a2c19a6205d1f62cbbc27a9db262f44aaa6c2/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/962726196b07d833abcad4971ae5c8e59ce6e9e33cea3f17dfaac6015a358ba7/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/787ed521e548760bcda858a6812a1ab1245e7860b4d30aa077d3123ff02677af/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a70f82dab7bed1232fda417e2598fd1cc8f444b890daf8e23bd93155ec0e7a7f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4387096a1f0d29f59d94b5f78ec5450a95bc7aa715a619d30d445144bd150301/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/94f3fdcb838368c8cedeb379a8e551b2b6bd521dd03315bf576174a0273e921a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bd6c08584e55416d6d556973920909defba50435e54f1928a74fc66ef8891549/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.307946 4484 manager.go:217] Machine: {Timestamp:2026-11-01 12:22:10.307500128 +0000 UTC m=+0.057443097 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099906 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d54b96f7c04ca6d40b32dfd16fd5a SystemUUID:ec2d54b9-6f7c-04ca-6d40-b32dfd16fd5a BootID:526e5d3e-0ef3-402c-b16c-be243405f14b Filesystems:[{Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec82d06036b0594279ea05327b3bf50199370b5b7b1d875c852b7f7391de8eaf/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e49cbf8c1a286e412d409b37bf338372942ba75209c6294c1138ec182ed698e/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~projected/kube-api-access-fc782 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a950d3512826df96b108ec836c00a68e296cb0addfd5f32896ced11913bea537/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b563c5669d10d6534b6a9276457322fd80cb5b4c4c7715b711aa1ceaa83576a0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b/volumes/kubernetes.io~projected/kube-api-access-zlzlz DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/129138ad-2f7a-4994-8693-61c7b660f53f/volumes/kubernetes.io~projected/kube-api-access-xgltg DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6/volumes/kubernetes.io~projected/kube-api-access-5btj8 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/825516f7-6409-4911-835f-7e9abb4177b5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9e7bd03-73c7-4c34-b0d6-102d25d269b4/volumes/kubernetes.io~projected/kube-api-access-zzmvr DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volumes/kubernetes.io~projected/kube-api-access-vkcpc DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/825516f7-6409-4911-835f-7e9abb4177b5/volumes/kubernetes.io~projected/kube-api-access-txtwh DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1e8956c-23c1-4d94-8574-62fb54e26e9f/volumes/kubernetes.io~projected/kube-api-access-prg5z DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64792f549b0bce70a894d959ae3359f5b0853b880c049a2d7521f0e41ad51ccd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/678c9f2a-d55a-4f62-913c-650cbf020460/volumes/kubernetes.io~projected/kube-api-access-zb8h6 DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4138765e71cbba52cb85d5a0c8233412264a10e4c1f40a6eb42bb1df9720c483/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/743d7eb98c96d2f19fbdcf16f2a1d3f4a8788a984b389c6fdd65316032b97691/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c5e66e40-bbf7-4a34-848c-0f2e7842b71f/volumes/kubernetes.io~projected/kube-api-access-85crl DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81c2246b52ec464dd91e1c03957da96d01d1593ce776b62b93c4149cd8f1e398/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a422fe5-8a7c-482a-9564-56151c0ca16c/volumes/kubernetes.io~projected/kube-api-access-69p6p DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51e6e124-97cb-448a-a1a3-27ad1f617d87/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9170b98-0937-44e0-86d9-73821df95fef/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbc0e0ae1f4835de538a5acce6f72c8ee74ef116f11c2d2bfc5aeb331dd9694c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51e6e124-97cb-448a-a1a3-27ad1f617d87/volumes/kubernetes.io~projected/kube-api-access-qs5f6 DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5b85d1-29a8-459e-86df-38b27c994538/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e736662491a170ff72b02c67db3596533026d58ff14f5d5eb5f339adff754081/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb547296-0fb6-4120-98ad-31e2f082eaca/volumes/kubernetes.io~projected/kube-api-access-vjbt5 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee10a231-11cb-42d1-971f-08ebea7d040c/volumes/kubernetes.io~projected/kube-api-access-5ch78 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2f171cc-8de9-408d-9e4e-b9b20247048a/volumes/kubernetes.io~projected/kube-api-access-9qw5h DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3553697171f4935b56e1935a782790eca6ae5560a5ab4b6f579c8ce65324898e/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f89016e51efa97d150503cf8b0009486f761d310de94d71d5a247e57e0f99897/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52417111-186c-4db8-9b3b-3b0315901dc2/volumes/kubernetes.io~projected/kube-api-access-drpvd DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8153630648b75aeb222afb7df4585bb2566cb9b9405a7bc08399547f308f36b6/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3553697171f4935 MacAddress:b2:32:0c:56:07:69 Speed:10000 Mtu:8901} {Name:6e49cbf8c1a286e MacAddress:32:50:3a:78:01:64 Speed:10000 Mtu:8901} {Name:8153630648b75ae MacAddress:4e:c2:b7:2b:67:67 Speed:10000 Mtu:8901} {Name:9b7711bdb742154 MacAddress:de:ad:1f:de:0e:eb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ef:b4:85:79 Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:ee:7f:8a:21:15 Speed:0 Mtu:8901} {Name:e736662491a170f MacAddress:46:ee:4d:69:9a:ac Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ef:b4:85:79 Speed:0 Mtu:9001} {Name:f89016e51efa97d MacAddress:c6:21:09:88:b6:94 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:06:85:e2:27:5c:ff Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:27:98:37:d9:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308214 4484 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308254 4484 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308578 4484 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308603 4484 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-174.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308723 4484 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308730 4484 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308745 4484 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308765 4484 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.308828 4484 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309023 4484 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309133 4484 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309164 4484 kubelet.go:493] "Attempting to sync node with API server" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309177 4484 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309193 4484 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309201 4484 kubelet.go:405] "Adding apiserver pod source" Nov 01 12:22:10.309744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.309208 4484 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 12:22:10.310258 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.310136 4484 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 12:22:10.310258 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.310152 4484 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 12:22:10.310685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.310668 4484 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 12:22:10.321420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321404 4484 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 12:22:10.321420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321417 4484 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 12:22:10.321534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321527 4484 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 12:22:10.321737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321725 4484 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 12:22:10.321766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321760 4484 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 12:22:10.321874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321865 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 12:22:10.321906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321879 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 12:22:10.321906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321885 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 12:22:10.321906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321891 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 12:22:10.321906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321897 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 12:22:10.321906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321903 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321912 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321921 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321930 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321936 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321947 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321963 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321987 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 12:22:10.322034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.321995 4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 12:22:10.325459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.325446 4484 server.go:1263] "Started kubelet" Nov 01 12:22:10.325591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.325540 4484 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 12:22:10.325643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.325547 4484 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 12:22:10.325643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.325630 4484 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 12:22:10.326008 ip-10-0-140-174 systemd[1]: Started Kubernetes Kubelet. Nov 01 12:22:10.327172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.326941 4484 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 12:22:10.327529 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.327501 4484 server.go:317] "Adding debug handlers to kubelet server" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330633 4484 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330641 4484 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330700 4484 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:32 +0000 UTC" deadline="2028-06-13 23:45:26.18383494 +0000 UTC" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330716 4484 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14171h23m15.853121536s" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330741 4484 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330753 4484 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.330824 4484 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 12:22:10.331143 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.330823 4484 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332353 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5b85d1-29a8-459e-86df-38b27c994538" volumeName="kubernetes.io/projected/4b5b85d1-29a8-459e-86df-38b27c994538-kube-api-access-fc782" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332388 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c3a8cd4-87b7-4dce-9109-6a3e40960f4c" volumeName="kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-env-overrides" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332402 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51e6e124-97cb-448a-a1a3-27ad1f617d87" volumeName="kubernetes.io/configmap/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-cabundle" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332414 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5e66e40-bbf7-4a34-848c-0f2e7842b71f" volumeName="kubernetes.io/projected/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-kube-api-access-85crl" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332426 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" volumeName="kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332437 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" volumeName="kubernetes.io/projected/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-kube-api-access-zzmvr" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332449 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="129138ad-2f7a-4994-8693-61c7b660f53f" volumeName="kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332461 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="678c9f2a-d55a-4f62-913c-650cbf020460" volumeName="kubernetes.io/configmap/678c9f2a-d55a-4f62-913c-650cbf020460-serviceca" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332475 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="825516f7-6409-4911-835f-7e9abb4177b5" volumeName="kubernetes.io/projected/825516f7-6409-4911-835f-7e9abb4177b5-kube-api-access-txtwh" seLinuxMountContext="" Nov 01 12:22:10.332483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332488 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f171cc-8de9-408d-9e4e-b9b20247048a" volumeName="kubernetes.io/empty-dir/b2f171cc-8de9-408d-9e4e-b9b20247048a-tmp-dir" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332499 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e8956c-23c1-4d94-8574-62fb54e26e9f" volumeName="kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cni-binary-copy" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332509 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c3a8cd4-87b7-4dce-9109-6a3e40960f4c" volumeName="kubernetes.io/secret/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332520 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9170b98-0937-44e0-86d9-73821df95fef" volumeName="kubernetes.io/configmap/a9170b98-0937-44e0-86d9-73821df95fef-konnectivity-ca" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332550 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e8956c-23c1-4d94-8574-62fb54e26e9f" volumeName="kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-daemon-config" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332570 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a919c6-24a0-40a2-96b9-ea4e7c7098a6" volumeName="kubernetes.io/projected/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-kube-api-access-5btj8" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332586 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b" volumeName="kubernetes.io/projected/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-api-access-zlzlz" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332601 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5b85d1-29a8-459e-86df-38b27c994538" volumeName="kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-etc-tuned" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332614 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c3a8cd4-87b7-4dce-9109-6a3e40960f4c" volumeName="kubernetes.io/projected/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-kube-api-access-vkcpc" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332626 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="825516f7-6409-4911-835f-7e9abb4177b5" volumeName="kubernetes.io/secret/825516f7-6409-4911-835f-7e9abb4177b5-cert" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332637 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f171cc-8de9-408d-9e4e-b9b20247048a" volumeName="kubernetes.io/configmap/b2f171cc-8de9-408d-9e4e-b9b20247048a-config-volume" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332648 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f171cc-8de9-408d-9e4e-b9b20247048a" volumeName="kubernetes.io/projected/b2f171cc-8de9-408d-9e4e-b9b20247048a-kube-api-access-9qw5h" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332662 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" volumeName="kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-binary-copy" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332675 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f93d1106-ec10-4183-b91c-bdbf0ea5fa9a" volumeName="kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332686 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb547296-0fb6-4120-98ad-31e2f082eaca" volumeName="kubernetes.io/empty-dir/fb547296-0fb6-4120-98ad-31e2f082eaca-tmp-dir" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332699 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="129138ad-2f7a-4994-8693-61c7b660f53f" volumeName="kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332711 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="129138ad-2f7a-4994-8693-61c7b660f53f" volumeName="kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332725 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c3a8cd4-87b7-4dce-9109-6a3e40960f4c" volumeName="kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-config" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332738 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" volumeName="kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332750 4484 factory.go:55] Registering systemd factory Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332756 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b" volumeName="kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332766 4484 factory.go:223] Registration of the systemd container factory successfully Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332770 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb547296-0fb6-4120-98ad-31e2f082eaca" volumeName="kubernetes.io/projected/fb547296-0fb6-4120-98ad-31e2f082eaca-kube-api-access-vjbt5" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332781 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5b85d1-29a8-459e-86df-38b27c994538" volumeName="kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-tmp" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332794 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c3a8cd4-87b7-4dce-9109-6a3e40960f4c" volumeName="kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-script-lib" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332807 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51e6e124-97cb-448a-a1a3-27ad1f617d87" volumeName="kubernetes.io/projected/51e6e124-97cb-448a-a1a3-27ad1f617d87-kube-api-access-qs5f6" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332820 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="678c9f2a-d55a-4f62-913c-650cbf020460" volumeName="kubernetes.io/projected/678c9f2a-d55a-4f62-913c-650cbf020460-kube-api-access-zb8h6" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332832 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e8956c-23c1-4d94-8574-62fb54e26e9f" volumeName="kubernetes.io/projected/c1e8956c-23c1-4d94-8574-62fb54e26e9f-kube-api-access-prg5z" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332845 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b" volumeName="kubernetes.io/empty-dir/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-data-volume" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332856 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a422fe5-8a7c-482a-9564-56151c0ca16c" volumeName="kubernetes.io/projected/2a422fe5-8a7c-482a-9564-56151c0ca16c-kube-api-access-69p6p" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332869 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51e6e124-97cb-448a-a1a3-27ad1f617d87" volumeName="kubernetes.io/secret/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-key" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332880 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9170b98-0937-44e0-86d9-73821df95fef" volumeName="kubernetes.io/secret/a9170b98-0937-44e0-86d9-73821df95fef-agent-certs" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332894 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="129138ad-2f7a-4994-8693-61c7b660f53f" volumeName="kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332905 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52417111-186c-4db8-9b3b-3b0315901dc2" volumeName="kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332916 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a919c6-24a0-40a2-96b9-ea4e7c7098a6" volumeName="kubernetes.io/configmap/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-iptables-alerter-script" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332929 4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee10a231-11cb-42d1-971f-08ebea7d040c" volumeName="kubernetes.io/projected/ee10a231-11cb-42d1-971f-08ebea7d040c-kube-api-access-5ch78" seLinuxMountContext="" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332953 4484 reconstruct.go:97] "Volume reconstruction finished" Nov 01 12:22:10.332998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.332961 4484 reconciler.go:29] "Reconciler: start to sync state" Nov 01 12:22:10.334798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.333044 4484 factory.go:153] Registering CRI-O factory Nov 01 12:22:10.334798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.333061 4484 factory.go:223] Registration of the crio container factory successfully Nov 01 12:22:10.334798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.333112 4484 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 12:22:10.334798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.333130 4484 factory.go:103] Registering Raw factory Nov 01 12:22:10.334798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.333145 4484 manager.go:1199] Started watching for new ooms in manager Nov 01 12:22:10.334798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.333516 4484 manager.go:319] Starting recovery of all containers Nov 01 12:22:10.347394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.347337 4484 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 12:22:10.361167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.361131 4484 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 12:22:10.361167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.361163 4484 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 12:22:10.361368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.361183 4484 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 12:22:10.361368 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.361222 4484 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 12:22:10.369840 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.369809 4484 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="c610174ce4a5d8138365ebd8a611cc8faddaf3f71f4bc7718e8abcd02dbe2758" exitCode=0 Nov 01 12:22:10.369921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.369843 4484 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="220fcb7356a284bc9e43dba86f136187fdec2cce2ef1df50c9ac6f6e6416ac94" exitCode=0 Nov 01 12:22:10.369921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.369852 4484 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="782e744be06e173680177f4d007ec7d807d50d4b5655e7ee2b7664aadd3d35e8" exitCode=0 Nov 01 12:22:10.369921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.369861 4484 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="51762a5bbdb98389dd94d3cd8c6c4294a875c883154173d67534c1a17f54d209" exitCode=0 Nov 01 12:22:10.369921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.369868 4484 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="86e4c4c4ee4c81538d9cec825dda3335be0045f666c1d54690148ee8132b4d29" exitCode=0 Nov 01 12:22:10.369921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.369873 4484 generic.go:356] "Generic (PLEG): container finished" podID="c9e7bd03-73c7-4c34-b0d6-102d25d269b4" containerID="d5e66bb48c0f508aa278506e53cd7b58a470469b9cf35cef655779acc3c25c3c" exitCode=0 Nov 01 12:22:10.388979 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.388944 4484 generic.go:356] "Generic (PLEG): container finished" podID="1b76893291fa1dd017b75cc7b5ec1b7d" containerID="2bd02ee4d37a9714dc4e12595c7ca710b3783475c9b66478adf04a5a96b39c0c" exitCode=0 Nov 01 12:22:10.401604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.401583 4484 manager.go:324] Recovery completed Nov 01 12:22:10.414081 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.414061 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.415173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.415154 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.415263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.415186 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.415263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.415204 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.416531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416515 4484 cpu_manager.go:226] "Starting" policy="none" Nov 01 12:22:10.416531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416526 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.416655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416527 4484 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 12:22:10.416655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416601 4484 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 12:22:10.416805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416789 4484 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 12:22:10.416862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416805 4484 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 12:22:10.416862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416826 4484 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 12:22:10.416862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416835 4484 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 12:22:10.416862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416843 4484 policy_none.go:50] "Start" Nov 01 12:22:10.416862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416857 4484 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 12:22:10.417088 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.416870 4484 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 12:22:10.417569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.417555 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.417615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.417581 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.417615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.417592 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.417919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.417905 4484 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 12:22:10.417968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.417924 4484 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 12:22:10.417968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.417933 4484 policy_none.go:44] "Start" Nov 01 12:22:10.422025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.421999 4484 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 12:22:10.424961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.424943 4484 manager.go:341] "Starting Device Plugin manager" Nov 01 12:22:10.425052 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.424976 4484 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 12:22:10.425052 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.424990 4484 server.go:87] "Starting device plugin registration server" Nov 01 12:22:10.425378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425363 4484 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 12:22:10.425436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425381 4484 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 12:22:10.425585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425569 4484 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 12:22:10.425695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425662 4484 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 12:22:10.425747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425704 4484 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 12:22:10.425747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425712 4484 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 12:22:10.425891 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.425792 4484 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T12:22:10.425694877Z","UUID":"6ce8531e-1f90-42ee-8b9e-b6bdf5e649b9","Handler":null,"Name":"","Endpoint":""} Nov 01 12:22:10.427786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.427769 4484 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 12:22:10.427911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.427891 4484 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 12:22:10.428117 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.428094 4484 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 12:22:10.428178 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.428143 4484 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" Nov 01 12:22:10.462247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.462177 4484 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal"] Nov 01 12:22:10.462361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.462295 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.463290 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.463275 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.463389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.463324 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.463389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.463339 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.463613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.463602 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.463764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.463749 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.464384 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464370 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.464447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464393 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.464447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464408 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.464524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464469 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.464524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464488 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.464524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464498 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.464603 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.464529 4484 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.464756 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464730 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerStarted","Data":"711c66617c821acb9f5ff2708bb2787ef058af412a699f514add92166c211f5c"} Nov 01 12:22:10.464786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464763 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerDied","Data":"2bd02ee4d37a9714dc4e12595c7ca710b3783475c9b66478adf04a5a96b39c0c"} Nov 01 12:22:10.464786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464778 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"1b76893291fa1dd017b75cc7b5ec1b7d","Type":"ContainerStarted","Data":"bdd97a813dd52603f36516a3528a8baec5c363d8a4e45e4fc9f25155cd3f1bf7"} Nov 01 12:22:10.464843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464794 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"2c12124bf0630fd70bf94c8ced58f02e","Type":"ContainerStarted","Data":"bdc8ba644cbf02f75ebf41846d04eb3257fc8b693646022b94a68cfed33829f8"} Nov 01 12:22:10.464843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464805 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" event={"ID":"2c12124bf0630fd70bf94c8ced58f02e","Type":"ContainerStarted","Data":"f7571270d0bdfa5235c08fb6e41d1bd55beb0c2ee641e5dc55ce1b35a246cbb6"} Nov 01 12:22:10.464843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.464806 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.465657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.465644 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.465708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.465673 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.465708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.465685 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.465809 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.465798 4484 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-174.us-west-2.compute.internal\" not found" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.503399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.503379 4484 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:22:10.503478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.503400 4484 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:22:10.503514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.503500 4484 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 12:22:10.504227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.504209 4484 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:22:10.526333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.526315 4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 12:22:10.527909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.527894 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 12:22:10.527996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.527927 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 12:22:10.527996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.527944 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 12:22:10.527996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.527994 4484 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.531128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.531113 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.550497 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.550473 4484 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 12:22:10.591162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.591140 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 12:22:10.591249 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.591180 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.591249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.591198 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.604514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.604494 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c12124bf0630fd70bf94c8ced58f02e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"2c12124bf0630fd70bf94c8ced58f02e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.604582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.604520 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.604582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.604536 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.638893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.638870 4484 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.638991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.638931 4484 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.639480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.639465 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeNotReady" Nov 01 12:22:10.639562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.639498 4484 setters.go:546] "Node became not ready" node="ip-10-0-140-174.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T12:22:10Z","lastTransitionTime":"2026-11-01T12:22:10Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 12:22:10.652597 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.652571 4484 nodeinfomanager.go:471] existing CSINode "ip-10-0-140-174.us-west-2.compute.internal" is owned by different node (oldNodeID="14d14585-2e25-4bbb-97bd-130bc07ed3a5", newNodeID=""), cleaning up... Nov 01 12:22:10.687953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.687932 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 12:22:10.688035 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:10.687979 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.705017 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.704997 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.705121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.705022 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.705121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.705040 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c12124bf0630fd70bf94c8ced58f02e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"2c12124bf0630fd70bf94c8ced58f02e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.705121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.705079 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c12124bf0630fd70bf94c8ced58f02e-config\") pod \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"2c12124bf0630fd70bf94c8ced58f02e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.705121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.705110 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:10.705267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:10.705111 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b76893291fa1dd017b75cc7b5ec1b7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" (UID: \"1b76893291fa1dd017b75cc7b5ec1b7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:11.309889 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.309862 4484 apiserver.go:52] "Watching apiserver" Nov 01 12:22:11.401781 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.401756 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:11.401917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.401867 4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:11.478570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.478527 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 12:22:11.478570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.478530 4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 12:22:11.478854 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.478586 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:11.478854 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.478624 4484 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 12:22:11.478854 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.478633 4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal" Nov 01 12:22:11.479946 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.479811 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal","openshift-dns/dns-default-tv5nb","openshift-insights/insights-runtime-extractor-5q4xn","openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-174.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-7s548","openshift-insights/periodic-gathering-r5bbx-8fck5","openshift-network-diagnostics/network-check-target-56m2d","openshift-dns/node-resolver-k6w7r","openshift-image-registry/node-ca-kc9l9","openshift-multus/multus-v4qqn","openshift-ovn-kubernetes/ovnkube-node-jn9jc","openshift-service-ca/service-ca-5fffc97bdf-wgsgn","kube-system/konnectivity-agent-6kq7t","openshift-multus/network-metrics-daemon-t5ghs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss","openshift-network-operator/iptables-alerter-w6d8r","openshift-multus/multus-additional-cni-plugins-2dcr2","openshift-ingress-canary/ingress-canary-l64fw","kube-system/global-pull-secret-syncer-x8prg"] Nov 01 12:22:11.484212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.484194 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:11.485374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.485357 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.486489 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.486471 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.532668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.532636 4484 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 12:22:11.536332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536274 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 12:22:11.536332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536276 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9cl4q\"" Nov 01 12:22:11.536537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536351 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhblh\"" Nov 01 12:22:11.536537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536456 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 12:22:11.536537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536473 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z9pcl\"" Nov 01 12:22:11.536537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536477 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7twmh\"" Nov 01 12:22:11.536537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536500 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\"" Nov 01 12:22:11.536775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536566 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qrhfv\"" Nov 01 12:22:11.536775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536565 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhf7p\"" Nov 01 12:22:11.536775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536645 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 12:22:11.536775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536736 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 12:22:11.536775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536743 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2z6g\"" Nov 01 12:22:11.536929 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.536821 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfjvd\"" Nov 01 12:22:11.537072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537046 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsq77\"" Nov 01 12:22:11.537166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537105 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 12:22:11.537166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537117 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 12:22:11.537271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537209 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.537271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537242 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.537271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537242 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9t2fc\"" Nov 01 12:22:11.537403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537383 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.537516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537500 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 12:22:11.537551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537506 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 12:22:11.537667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537653 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.537667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537660 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537688 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537711 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537721 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537731 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537691 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537750 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 12:22:11.537775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537761 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537761 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537655 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537727 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537809 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537805 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537688 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537863 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537883 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537888 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537907 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537913 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 12:22:11.538045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.537982 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.553066 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.553049 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9qzg6\"" Nov 01 12:22:11.553413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.553397 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.579638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.579564 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.579638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.579586 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 12:22:11.579815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.579662 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.579815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.579690 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jtrgb\"" Nov 01 12:22:11.581200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.581184 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.581249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.581208 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-svx22\"" Nov 01 12:22:11.581289 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.581260 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.581939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.581922 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 12:22:11.581996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.581953 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 12:22:11.582043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.582023 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 12:22:11.582336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.582323 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.584327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.584173 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 12:22:11.584954 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.584671 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-86gmg\"" Nov 01 12:22:11.584954 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.584915 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 12:22:11.585205 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.585187 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 12:22:11.585268 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.585207 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-thbc5\"" Nov 01 12:22:11.585418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.585392 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 12:22:11.585578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.585563 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 12:22:11.585816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.585803 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lt5kb\"" Nov 01 12:22:11.585927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.585913 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 12:22:11.588324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.588293 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 12:22:11.610871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.610848 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.610955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.610876 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cnibin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.610955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.610892 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-var-lib-kubelet\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.610955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.610911 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-host\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.611059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.610963 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:11.611059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.610986 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-registration-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.611059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611006 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8h6\" (UniqueName: \"kubernetes.io/projected/678c9f2a-d55a-4f62-913c-650cbf020460-kube-api-access-zb8h6\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.611148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611045 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.611243 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611220 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:11.611327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611276 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-host-slash\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.611383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611338 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-bin\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.611383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611367 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-crio-socket\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.611462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611385 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9170b98-0937-44e0-86d9-73821df95fef-agent-certs\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:22:11.611462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611431 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-conf\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.611542 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611495 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:11.611583 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611544 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.611636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611618 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.611669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611641 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-var-lib-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.611711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611680 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-daemon-config\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.611758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611709 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-multus-certs\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.611758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611709 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9170b98-0937-44e0-86d9-73821df95fef-agent-certs\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:22:11.611758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611736 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/678c9f2a-d55a-4f62-913c-650cbf020460-serviceca\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.611892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611780 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-original-pull-secret\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.611892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611802 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-kubelet\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.611892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611837 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.611892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611869 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611893 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/678c9f2a-d55a-4f62-913c-650cbf020460-serviceca\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611900 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-etc-kubernetes\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611921 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cnibin\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611939 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678c9f2a-d55a-4f62-913c-650cbf020460-host\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611960 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611963 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-daemon-config\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.611979 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-netd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612002 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-tmp\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612026 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc782\" (UniqueName: \"kubernetes.io/projected/4b5b85d1-29a8-459e-86df-38b27c994538-kube-api-access-fc782\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.612080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612050 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612086 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-device-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612104 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ch78\" (UniqueName: \"kubernetes.io/projected/ee10a231-11cb-42d1-971f-08ebea7d040c-kube-api-access-5ch78\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612088 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-tmp\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612133 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612173 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-node-log\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612207 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612235 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-k8s-cni-cncf-io\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612270 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prg5z\" (UniqueName: \"kubernetes.io/projected/c1e8956c-23c1-4d94-8574-62fb54e26e9f-kube-api-access-prg5z\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612323 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9170b98-0937-44e0-86d9-73821df95fef-konnectivity-ca\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612350 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-key\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612384 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btj8\" (UniqueName: \"kubernetes.io/projected/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-kube-api-access-5btj8\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612493 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-system-cni-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.612534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612522 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-modprobe-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612544 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9170b98-0937-44e0-86d9-73821df95fef-konnectivity-ca\") pod \"konnectivity-agent-6kq7t\" (UID: \"a9170b98-0937-44e0-86d9-73821df95fef\") " pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612571 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-kubernetes\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612596 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-run\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612609 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/51e6e124-97cb-448a-a1a3-27ad1f617d87-signing-key\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612655 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-etc-tuned\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612689 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85crl\" (UniqueName: \"kubernetes.io/projected/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-kube-api-access-85crl\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612724 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612741 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b5b85d1-29a8-459e-86df-38b27c994538-etc-tuned\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612749 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-systemd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612773 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-hostroot\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612802 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f171cc-8de9-408d-9e4e-b9b20247048a-config-volume\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612827 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-sys-fs\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612849 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-dbus\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-iptables-alerter-script\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612918 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-socket-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612941 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-env-overrides\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612956 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-data-volume\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.612982 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613009 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f171cc-8de9-408d-9e4e-b9b20247048a-config-volume\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613051 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-data-volume\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613068 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-conf-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.613092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613095 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-kubelet-config\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613121 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-systemd-units\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613126 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-env-overrides\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613159 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613164 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-etc-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613176 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-iptables-alerter-script\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613200 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-config\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613236 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-script-lib\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613256 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613272 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-sys\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613350 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-netns\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613364 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-ovn\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613386 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcpc\" (UniqueName: \"kubernetes.io/projected/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-kube-api-access-vkcpc\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613444 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613453 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-config\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613495 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzlz\" (UniqueName: \"kubernetes.io/projected/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-api-access-zlzlz\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613543 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovnkube-script-lib\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613552 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-kubelet\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613641 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613683 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613737 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-systemd\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613769 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613783 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613806 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.613945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.613831 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69p6p\" (UniqueName: \"kubernetes.io/projected/2a422fe5-8a7c-482a-9564-56151c0ca16c-kube-api-access-69p6p\") pod \"migrator-66df5d496b-24rdd\" (UID: \"2a422fe5-8a7c-482a-9564-56151c0ca16c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614093 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-multus\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614113 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb547296-0fb6-4120-98ad-31e2f082eaca-hosts-file\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614130 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbt5\" (UniqueName: \"kubernetes.io/projected/fb547296-0fb6-4120-98ad-31e2f082eaca-kube-api-access-vjbt5\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614146 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw5h\" (UniqueName: \"kubernetes.io/projected/b2f171cc-8de9-408d-9e4e-b9b20247048a-kube-api-access-9qw5h\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614183 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-slash\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614277 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-log-socket\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614296 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-socket-dir-parent\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614327 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-bin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614344 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmvr\" (UniqueName: \"kubernetes.io/projected/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-kube-api-access-zzmvr\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614484 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysconfig\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614514 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb547296-0fb6-4120-98ad-31e2f082eaca-tmp-dir\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614539 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825516f7-6409-4911-835f-7e9abb4177b5-cert\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614576 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtwh\" (UniqueName: \"kubernetes.io/projected/825516f7-6409-4911-835f-7e9abb4177b5-kube-api-access-txtwh\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614650 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614658 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb547296-0fb6-4120-98ad-31e2f082eaca-tmp-dir\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614670 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5f6\" (UniqueName: \"kubernetes.io/projected/51e6e124-97cb-448a-a1a3-27ad1f617d87-kube-api-access-qs5f6\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614785 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-netns\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614815 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614839 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614871 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f171cc-8de9-408d-9e4e-b9b20247048a-tmp-dir\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.614893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614889 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825516f7-6409-4911-835f-7e9abb4177b5-cert\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614928 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f171cc-8de9-408d-9e4e-b9b20247048a-tmp-dir\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.614998 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-system-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615004 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615086 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cni-binary-copy\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615095 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615118 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-lib-modules\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615175 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovn-node-metrics-cert\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615249 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cni-binary-copy\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.615459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.615359 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-ovn-node-metrics-cert\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.662993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.662966 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8h6\" (UniqueName: \"kubernetes.io/projected/678c9f2a-d55a-4f62-913c-650cbf020460-kube-api-access-zb8h6\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.662993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.662993 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpvd\" (UniqueName: \"kubernetes.io/projected/52417111-186c-4db8-9b3b-3b0315901dc2-kube-api-access-drpvd\") pod \"network-check-target-56m2d\" (UID: \"52417111-186c-4db8-9b3b-3b0315901dc2\") " pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:11.663286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.663268 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ch78\" (UniqueName: \"kubernetes.io/projected/ee10a231-11cb-42d1-971f-08ebea7d040c-kube-api-access-5ch78\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.664169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.664143 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85crl\" (UniqueName: \"kubernetes.io/projected/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-kube-api-access-85crl\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:11.664248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.664193 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc782\" (UniqueName: \"kubernetes.io/projected/4b5b85d1-29a8-459e-86df-38b27c994538-kube-api-access-fc782\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.664311 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.664282 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btj8\" (UniqueName: \"kubernetes.io/projected/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-kube-api-access-5btj8\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.664343 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.664327 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prg5z\" (UniqueName: \"kubernetes.io/projected/c1e8956c-23c1-4d94-8574-62fb54e26e9f-kube-api-access-prg5z\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.665110 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.665093 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcpc\" (UniqueName: \"kubernetes.io/projected/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-kube-api-access-vkcpc\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.666374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.666346 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69p6p\" (UniqueName: \"kubernetes.io/projected/2a422fe5-8a7c-482a-9564-56151c0ca16c-kube-api-access-69p6p\") pod \"migrator-66df5d496b-24rdd\" (UID: \"2a422fe5-8a7c-482a-9564-56151c0ca16c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-24rdd" Nov 01 12:22:11.666736 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.666714 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbt5\" (UniqueName: \"kubernetes.io/projected/fb547296-0fb6-4120-98ad-31e2f082eaca-kube-api-access-vjbt5\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.666805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.666722 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw5h\" (UniqueName: \"kubernetes.io/projected/b2f171cc-8de9-408d-9e4e-b9b20247048a-kube-api-access-9qw5h\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.667112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.667084 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtwh\" (UniqueName: \"kubernetes.io/projected/825516f7-6409-4911-835f-7e9abb4177b5-kube-api-access-txtwh\") pod \"ingress-canary-l64fw\" (UID: \"825516f7-6409-4911-835f-7e9abb4177b5\") " pod="openshift-ingress-canary/ingress-canary-l64fw" Nov 01 12:22:11.667265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.667243 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5f6\" (UniqueName: \"kubernetes.io/projected/51e6e124-97cb-448a-a1a3-27ad1f617d87-kube-api-access-qs5f6\") pod \"service-ca-5fffc97bdf-wgsgn\" (UID: \"51e6e124-97cb-448a-a1a3-27ad1f617d87\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" Nov 01 12:22:11.667347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.667266 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzlz\" (UniqueName: \"kubernetes.io/projected/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-kube-api-access-zlzlz\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.667415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.667344 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmvr\" (UniqueName: \"kubernetes.io/projected/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-kube-api-access-zzmvr\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.667616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.667598 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") pod \"periodic-gathering-r5bbx-8fck5\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:22:11.716106 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716065 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678c9f2a-d55a-4f62-913c-650cbf020460-host\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.716106 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716105 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716123 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-netd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716139 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-device-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-node-log\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716175 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/678c9f2a-d55a-4f62-913c-650cbf020460-host\") pod \"node-ca-kc9l9\" (UID: \"678c9f2a-d55a-4f62-913c-650cbf020460\") " pod="openshift-image-registry/node-ca-kc9l9" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716184 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716200 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-netd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716210 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-k8s-cni-cncf-io\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716208 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716229 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-node-log\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716231 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-device-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716243 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716248 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-system-cni-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716256 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-k8s-cni-cncf-io\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716279 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-system-cni-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716282 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-modprobe-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716322 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-kubernetes\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.716353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716344 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-run\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716369 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716381 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-kubernetes\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716397 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-systemd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716430 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-run\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716438 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-hostroot\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716445 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716414 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-modprobe-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716462 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-sys-fs\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716467 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-systemd\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716486 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-hostroot\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716488 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-dbus\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716514 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-socket-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716527 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-dbus\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716531 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-conf-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716536 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-sys-fs\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716548 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-kubelet-config\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716568 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-conf-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716592 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-socket-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-systemd-units\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716642 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f93d1106-ec10-4183-b91c-bdbf0ea5fa9a-kubelet-config\") pod \"global-pull-secret-syncer-x8prg\" (UID: \"f93d1106-ec10-4183-b91c-bdbf0ea5fa9a\") " pod="kube-system/global-pull-secret-syncer-x8prg" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716662 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-etc-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716674 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-systemd-units\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716679 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716705 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-sys\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716724 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-etc-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716731 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-netns\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716757 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-netns\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716757 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-sys\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716707 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716756 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-ovn\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716790 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-ovn\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716794 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-kubelet\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716820 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716836 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-systemd\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-kubelet\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716852 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716874 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-multus\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716898 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb547296-0fb6-4120-98ad-31e2f082eaca-hosts-file\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716915 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-systemd\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716922 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-slash\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716943 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-d\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716946 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-multus\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716954 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-log-socket\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716956 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-slash\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716982 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-socket-dir-parent\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.716985 4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716998 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-bin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716999 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb547296-0fb6-4120-98ad-31e2f082eaca-hosts-file\") pod \"node-resolver-k6w7r\" (UID: \"fb547296-0fb6-4120-98ad-31e2f082eaca\") " pod="openshift-dns/node-resolver-k6w7r" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717013 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysconfig\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.716985 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-log-socket\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717045 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-netns\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.717032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717054 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-var-lib-cni-bin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717083 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-socket-dir-parent\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717071 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysconfig\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.717086 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:12.217064078 +0000 UTC m=+1.967007051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717113 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717137 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717155 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-system-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717094 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-netns\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717170 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-lib-modules\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.717195 4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717210 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717229 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-system-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717236 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cnibin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.717242 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:12.217231385 +0000 UTC m=+1.967174345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : secret "metrics-daemon-secret" not found Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717240 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-lib-modules\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.717258 4484 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717271 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-multus-cni-dir\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717278 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-cnibin\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717286 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-var-lib-kubelet\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:11.717330 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:12.21731667 +0000 UTC m=+1.967259642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717343 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-host\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717362 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-registration-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717388 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-host-slash\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717413 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-bin\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717417 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee10a231-11cb-42d1-971f-08ebea7d040c-registration-dir\") pod \"aws-ebs-csi-driver-node-b8tss\" (UID: \"ee10a231-11cb-42d1-971f-08ebea7d040c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8tss" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717413 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-host\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717360 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-var-lib-kubelet\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717444 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2a919c6-24a0-40a2-96b9-ea4e7c7098a6-host-slash\") pod \"iptables-alerter-w6d8r\" (UID: \"e2a919c6-24a0-40a2-96b9-ea4e7c7098a6\") " pod="openshift-network-operator/iptables-alerter-w6d8r" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717449 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-cni-bin\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717467 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-crio-socket\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717483 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-conf\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717497 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-var-lib-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717516 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-crio-socket\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717528 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-multus-certs\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717531 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-var-lib-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717547 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-kubelet\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717562 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717576 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717598 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-etc-kubernetes\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717606 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-run-openvswitch\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717611 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-kubelet\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717621 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cnibin\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717631 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c3a8cd4-87b7-4dce-9109-6a3e40960f4c-host-run-ovn-kubernetes\") pod \"ovnkube-node-jn9jc\" (UID: \"4c3a8cd4-87b7-4dce-9109-6a3e40960f4c\") " pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717643 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-etc-kubernetes\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717612 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b5b85d1-29a8-459e-86df-38b27c994538-etc-sysctl-conf\") pod \"tuned-7s548\" (UID: \"4b5b85d1-29a8-459e-86df-38b27c994538\") " pod="openshift-cluster-node-tuning-operator/tuned-7s548" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717653 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9e7bd03-73c7-4c34-b0d6-102d25d269b4-cnibin\") pod \"multus-additional-cni-plugins-2dcr2\" (UID: \"c9e7bd03-73c7-4c34-b0d6-102d25d269b4\") " pod="openshift-multus/multus-additional-cni-plugins-2dcr2" Nov 01 12:22:11.718637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.717653 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c1e8956c-23c1-4d94-8574-62fb54e26e9f-host-run-multus-certs\") pod \"multus-v4qqn\" (UID: \"c1e8956c-23c1-4d94-8574-62fb54e26e9f\") " pod="openshift-multus/multus-v4qqn" Nov 01 12:22:11.743390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.743364 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:11.788941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:11.788925 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:22:12.221469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:12.221430 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:12.221673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:12.221479 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:12.221673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:12.221516 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:12.221673 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:12.221609 4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:12.221673 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:12.221659 4484 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:12.221673 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:12.221668 4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 12:22:12.221871 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:12.221693 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:13.221673483 +0000 UTC m=+2.971616453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:12.221871 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:12.221724 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:13.221707635 +0000 UTC m=+2.971650591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:12.221871 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:12.221768 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:13.221756398 +0000 UTC m=+2.971699357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : secret "metrics-daemon-secret" not found Nov 01 12:22:12.647656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:12.647625 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:12.750833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:12.750803 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:13.229873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:13.229838 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:13.229873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:13.229883 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:13.230113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:13.229957 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:13.230113 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:13.230005 4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 12:22:13.230113 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:13.230067 4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:13.230113 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:13.230089 4484 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:13.230113 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:13.230104 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:15.230082177 +0000 UTC m=+4.980025153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : secret "metrics-daemon-secret" not found Nov 01 12:22:13.230353 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:13.230128 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:15.230112897 +0000 UTC m=+4.980055854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:13.230353 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:13.230182 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:15.230168344 +0000 UTC m=+4.980111315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:13.332866 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:13.332840 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k6w7r_fb547296-0fb6-4120-98ad-31e2f082eaca/dns-node-resolver/0.log" Nov 01 12:22:13.429883 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:13.429849 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:13.430045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:13.430034 4484 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 12:22:14.278817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:14.278779 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:14.390990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:14.390953 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:14.535747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:14.535729 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kc9l9_678c9f2a-d55a-4f62-913c-650cbf020460/node-ca/0.log" Nov 01 12:22:14.899166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:14.899082 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:15.132270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.132238 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l64fw_825516f7-6409-4911-835f-7e9abb4177b5/serve-healthcheck-canary/0.log" Nov 01 12:22:15.246729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.246692 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:15.246898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.246737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:15.246898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.246758 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:15.246898 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:15.246858 4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:15.246898 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:15.246875 4484 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:15.247032 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:15.246859 4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 12:22:15.247032 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:15.246923 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:19.246910226 +0000 UTC m=+8.996853196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:15.247032 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:15.246937 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:19.246929945 +0000 UTC m=+8.996872916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:15.247032 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:15.246951 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs podName:c5e66e40-bbf7-4a34-848c-0f2e7842b71f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:19.246943123 +0000 UTC m=+8.996886083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs") pod "network-metrics-daemon-t5ghs" (UID: "c5e66e40-bbf7-4a34-848c-0f2e7842b71f") : secret "metrics-daemon-secret" not found Nov 01 12:22:15.413075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.413031 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wgsgn" event={"ID":"51e6e124-97cb-448a-a1a3-27ad1f617d87","Type":"ContainerStarted","Data":"3d23a35ff49b6c5043123aed23eedc1ed949a0d253f7a159371eed5597c2e10a"} Nov 01 12:22:15.532999 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.532923 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-24rdd_2a422fe5-8a7c-482a-9564-56151c0ca16c/migrator/0.log" Nov 01 12:22:15.695804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:15.695766 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:16.185899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:16.185864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:16.296850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:16.296822 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:16.409085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:16.409035 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:16.530818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:16.530788 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:17.731029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:17.730992 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:17.933975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:17.933925 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:18.132655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:18.132533 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:18.333700 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:18.333637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:18.447005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:18.446937 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:18.560039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:18.560007 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:18.668735 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:18.668704 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:19.276606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.276564 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:19.276606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.276607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:19.277101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.276678 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:19.277101 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:19.276770 4484 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 12:22:19.277101 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:19.276785 4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 12:22:19.277101 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:19.276843 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls podName:b2f171cc-8de9-408d-9e4e-b9b20247048a nodeName:}" failed. No retries permitted until 2026-11-01 12:22:27.276826658 +0000 UTC m=+17.026769635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls") pod "dns-default-tv5nb" (UID: "b2f171cc-8de9-408d-9e4e-b9b20247048a") : secret "dns-default-metrics-tls" not found Nov 01 12:22:19.277101 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:22:19.276867 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls podName:ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b nodeName:}" failed. No retries permitted until 2026-11-01 12:22:27.276851327 +0000 UTC m=+17.026794298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q4xn" (UID: "ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b") : secret "insights-runtime-extractor-tls" not found Nov 01 12:22:19.277101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.276936 4484 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 12:22:19.278943 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.278923 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e66e40-bbf7-4a34-848c-0f2e7842b71f-metrics-certs\") pod \"network-metrics-daemon-t5ghs\" (UID: \"c5e66e40-bbf7-4a34-848c-0f2e7842b71f\") " pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:19.284888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.284867 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t5ghs" Nov 01 12:22:19.710119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.710045 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t5ghs"] Nov 01 12:22:19.714240 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:19.714211 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e66e40_bbf7_4a34_848c_0f2e7842b71f.slice/crio-7d38bea2efa5c589b31f71a616987185590946d5612d0d88a8953909e7d59c3c WatchSource:0}: Error finding container 7d38bea2efa5c589b31f71a616987185590946d5612d0d88a8953909e7d59c3c: Status 404 returned error can't find the container with id 7d38bea2efa5c589b31f71a616987185590946d5612d0d88a8953909e7d59c3c Nov 01 12:22:19.735308 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.735267 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:19.934372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:19.934338 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:20.133948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:20.133916 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:20.335649 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:20.335621 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:20.426688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:20.426600 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5ghs" event={"ID":"c5e66e40-bbf7-4a34-848c-0f2e7842b71f","Type":"ContainerStarted","Data":"7d38bea2efa5c589b31f71a616987185590946d5612d0d88a8953909e7d59c3c"} Nov 01 12:22:20.532123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:20.532090 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:20.801663 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:20.801633 4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-174.us-west-2.compute.internal" event="NodeReady" Nov 01 12:22:21.730918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:21.730888 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:21.934252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:21.934222 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:22.132120 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:22.132092 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:22.332951 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:22.332868 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:22.433692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:22.433657 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5ghs" event={"ID":"c5e66e40-bbf7-4a34-848c-0f2e7842b71f","Type":"ContainerStarted","Data":"aeb9102dfe853c478844e20f63693cdf59d12ae3622e1819e64ffd9b4dc51c3b"} Nov 01 12:22:22.433852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:22.433709 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t5ghs" event={"ID":"c5e66e40-bbf7-4a34-848c-0f2e7842b71f","Type":"ContainerStarted","Data":"d894febe43ad1a31b8b28c942afa30aadff5646bcc237dc76d76b4de88905c3b"} Nov 01 12:22:22.530733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:22.530699 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:23.392210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:23.392177 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:23.514001 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:23.513960 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:23.730742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:23.730699 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:23.933832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:23.933792 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:24.130788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:24.130708 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:24.237433 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:24.237404 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:24.380165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:24.380136 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:24.533046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:24.533007 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:25.346425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:25.346374 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:25.453240 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:25.453193 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:25.732402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:25.732368 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:25.933272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:25.933240 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:26.134763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:26.134687 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:26.332347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:26.332315 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:26.533770 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:26.533725 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:26.939439 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:26.939368 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6kq7t" Nov 01 12:22:27.336799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.336766 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:27.336799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.336801 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:27.339049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.339029 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f171cc-8de9-408d-9e4e-b9b20247048a-metrics-tls\") pod \"dns-default-tv5nb\" (UID: \"b2f171cc-8de9-408d-9e4e-b9b20247048a\") " pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:27.339145 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.339125 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q4xn\" (UID: \"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b\") " pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:27.386281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.386257 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:27.387351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.387335 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q4xn" Nov 01 12:22:27.810697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.810659 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tv5nb"] Nov 01 12:22:27.815401 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:27.815249 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f171cc_8de9_408d_9e4e_b9b20247048a.slice/crio-c36e27fdd339a341f8a9ebc9ee95d371090f6fee8be92efff6d850863e5cbcd2 WatchSource:0}: Error finding container c36e27fdd339a341f8a9ebc9ee95d371090f6fee8be92efff6d850863e5cbcd2: Status 404 returned error can't find the container with id c36e27fdd339a341f8a9ebc9ee95d371090f6fee8be92efff6d850863e5cbcd2 Nov 01 12:22:27.854221 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:27.854196 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2387a2_e468_4dd9_92fc_5a2f2ab67c9b.slice/crio-39e9686baa5fc6bd1d0f6f9fac57592a3d876b0345922f8cc68438d872461280 WatchSource:0}: Error finding container 39e9686baa5fc6bd1d0f6f9fac57592a3d876b0345922f8cc68438d872461280: Status 404 returned error can't find the container with id 39e9686baa5fc6bd1d0f6f9fac57592a3d876b0345922f8cc68438d872461280 Nov 01 12:22:27.856834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:27.856791 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5q4xn"] Nov 01 12:22:28.451445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:28.451409 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tv5nb" event={"ID":"b2f171cc-8de9-408d-9e4e-b9b20247048a","Type":"ContainerStarted","Data":"c36e27fdd339a341f8a9ebc9ee95d371090f6fee8be92efff6d850863e5cbcd2"} Nov 01 12:22:28.452999 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:28.452977 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q4xn" event={"ID":"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b","Type":"ContainerStarted","Data":"979e248e5b18694800deb7f10f1e7e8a781d75bc0b93fbd52ce3d13c1a2cbcd5"} Nov 01 12:22:28.453052 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:28.453024 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q4xn" event={"ID":"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b","Type":"ContainerStarted","Data":"39e9686baa5fc6bd1d0f6f9fac57592a3d876b0345922f8cc68438d872461280"} Nov 01 12:22:29.143393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.143356 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:29.149355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.149293 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bc77dddf-nsdgr"] Nov 01 12:22:29.174124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.174094 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249316 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50097a5b-2f81-46ee-85d0-c29a5991dce6-ca-trust-extracted\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249442 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-registry-tls\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249484 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50097a5b-2f81-46ee-85d0-c29a5991dce6-installation-pull-secrets\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249516 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50097a5b-2f81-46ee-85d0-c29a5991dce6-trusted-ca\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249543 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50097a5b-2f81-46ee-85d0-c29a5991dce6-image-registry-private-configuration\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249581 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50097a5b-2f81-46ee-85d0-c29a5991dce6-registry-certificates\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249606 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-bound-sa-token\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.249653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.249646 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkft\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-kube-api-access-kkkft\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.303055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.303016 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:29.330483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.330451 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 12:22:29.350129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350089 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-registry-tls\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350294 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350144 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50097a5b-2f81-46ee-85d0-c29a5991dce6-installation-pull-secrets\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350294 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350177 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50097a5b-2f81-46ee-85d0-c29a5991dce6-trusted-ca\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350323 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50097a5b-2f81-46ee-85d0-c29a5991dce6-image-registry-private-configuration\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350368 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50097a5b-2f81-46ee-85d0-c29a5991dce6-registry-certificates\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350457 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-bound-sa-token\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350511 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkft\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-kube-api-access-kkkft\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350555 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50097a5b-2f81-46ee-85d0-c29a5991dce6-ca-trust-extracted\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.350932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.350914 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50097a5b-2f81-46ee-85d0-c29a5991dce6-ca-trust-extracted\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.351043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.351027 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50097a5b-2f81-46ee-85d0-c29a5991dce6-registry-certificates\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.353282 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.353257 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50097a5b-2f81-46ee-85d0-c29a5991dce6-image-registry-private-configuration\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.356058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.356038 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 12:22:29.356879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.356863 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc77dddf-nsdgr"] Nov 01 12:22:29.361838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.361814 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50097a5b-2f81-46ee-85d0-c29a5991dce6-trusted-ca\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.364548 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.364529 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 12:22:29.364645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.364547 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqn6l\"" Nov 01 12:22:29.364645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.364625 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 12:22:29.373451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.373424 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-registry-tls\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.373719 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.373693 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50097a5b-2f81-46ee-85d0-c29a5991dce6-installation-pull-secrets\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.405588 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.405524 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-bound-sa-token\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.416455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.416432 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkft\" (UniqueName: \"kubernetes.io/projected/50097a5b-2f81-46ee-85d0-c29a5991dce6-kube-api-access-kkkft\") pod \"image-registry-6bc77dddf-nsdgr\" (UID: \"50097a5b-2f81-46ee-85d0-c29a5991dce6\") " pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.420249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.420209 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:29.503004 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.502973 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:29.681128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.681048 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:29.792029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:29.792002 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:30.133810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:30.133779 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc77dddf-nsdgr"] Nov 01 12:22:30.137272 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:30.137239 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50097a5b_2f81_46ee_85d0_c29a5991dce6.slice/crio-4c1a15db4cb6ff7a0e067132e108e1313a6abae606d2dc9d19183a7080a56231 WatchSource:0}: Error finding container 4c1a15db4cb6ff7a0e067132e108e1313a6abae606d2dc9d19183a7080a56231: Status 404 returned error can't find the container with id 4c1a15db4cb6ff7a0e067132e108e1313a6abae606d2dc9d19183a7080a56231 Nov 01 12:22:30.461382 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:30.461288 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q4xn" event={"ID":"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b","Type":"ContainerStarted","Data":"0a6728cc37366ae1a1ade632585ae0b84b4b1801709d842a987406aa275e6b3c"} Nov 01 12:22:30.462764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:30.462736 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" event={"ID":"50097a5b-2f81-46ee-85d0-c29a5991dce6","Type":"ContainerStarted","Data":"e33564fb3e9c1ba724efa42b44fa8ccf7222d387e1e55136f84917b0c478c282"} Nov 01 12:22:30.462873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:30.462782 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" event={"ID":"50097a5b-2f81-46ee-85d0-c29a5991dce6","Type":"ContainerStarted","Data":"4c1a15db4cb6ff7a0e067132e108e1313a6abae606d2dc9d19183a7080a56231"} Nov 01 12:22:30.462932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:30.462870 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:30.571405 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:30.571199 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" podStartSLOduration=1.5711828300000001 podStartE2EDuration="1.57118283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:30.570468453 +0000 UTC m=+20.320411430" watchObservedRunningTime="2026-11-01 12:22:30.57118283 +0000 UTC m=+20.321125838" Nov 01 12:22:31.467624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:31.467584 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tv5nb" event={"ID":"b2f171cc-8de9-408d-9e4e-b9b20247048a","Type":"ContainerStarted","Data":"843131748ea8d9c0fea51d8d0fa02ebf8155ab1d0bae897e3b6545aa44603710"} Nov 01 12:22:31.467792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:31.467680 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tv5nb" event={"ID":"b2f171cc-8de9-408d-9e4e-b9b20247048a","Type":"ContainerStarted","Data":"1520361e2b8f1195fb0ab65f61e1b07489360d9565ae1c7dfc9eab6657dd11d4"} Nov 01 12:22:31.467845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:31.467816 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:32.472915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:32.472822 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q4xn" event={"ID":"ef2387a2-e468-4dd9-92fc-5a2f2ab67c9b","Type":"ContainerStarted","Data":"982f31e3f473528529e2676947e490c4aa8c58dca0db0f8dd410d47dd605ff4c"} Nov 01 12:22:35.469230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:35.469190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:35.580412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:35.580369 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:35.718663 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:35.718631 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:35.829882 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:35.829853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:36.447605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:36.447577 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:36.692847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:36.692813 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:38.242180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:38.242154 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jn9jc" Nov 01 12:22:40.612355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:40.612285 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:40.723712 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:40.723668 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:40.833337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:40.833293 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:40.943782 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:40.943697 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:41.135861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:41.135832 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:41.474899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:41.474870 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tv5nb" Nov 01 12:22:41.602594 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:41.602559 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:41.718414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:41.718383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:41.839724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:41.839687 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:41.952450 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:41.952419 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:42.650230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:42.650204 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-56m2d" Nov 01 12:22:42.670421 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:42.670392 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:44.282757 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:44.282728 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:44.400359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:44.400330 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:44.539218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:44.539143 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:44.650514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:44.650482 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:44.760588 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:44.760560 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:46.349187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:46.349155 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:46.920212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:46.920174 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.078352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.078316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.330629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.330595 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.447004 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.446969 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.528420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.528386 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k"] Nov 01 12:22:47.532963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.532942 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.565607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.565582 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.568647 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.568623 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45qt\" (UniqueName: \"kubernetes.io/projected/cd430b13-1a61-4f19-8139-791ac71c1eff-kube-api-access-m45qt\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.568759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.568671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd430b13-1a61-4f19-8139-791ac71c1eff-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.568822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.568776 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd430b13-1a61-4f19-8139-791ac71c1eff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.568822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.568811 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd430b13-1a61-4f19-8139-791ac71c1eff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.628724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.628651 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 12:22:47.628724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.628661 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 12:22:47.628947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.628786 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 12:22:47.628947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.628900 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 12:22:47.628947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.628904 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f6nhv\"" Nov 01 12:22:47.629104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.628953 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 12:22:47.629722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.629705 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k"] Nov 01 12:22:47.669400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.669371 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45qt\" (UniqueName: \"kubernetes.io/projected/cd430b13-1a61-4f19-8139-791ac71c1eff-kube-api-access-m45qt\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.669546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.669409 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd430b13-1a61-4f19-8139-791ac71c1eff-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.669546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.669439 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd430b13-1a61-4f19-8139-791ac71c1eff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.669546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.669468 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd430b13-1a61-4f19-8139-791ac71c1eff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.670053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.670028 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd430b13-1a61-4f19-8139-791ac71c1eff-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.671792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.671774 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd430b13-1a61-4f19-8139-791ac71c1eff-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.671903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.671862 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd430b13-1a61-4f19-8139-791ac71c1eff-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.675787 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.675765 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l76dc"] Nov 01 12:22:47.678928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.678907 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.694481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.694459 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.724952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.724928 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45qt\" (UniqueName: \"kubernetes.io/projected/cd430b13-1a61-4f19-8139-791ac71c1eff-kube-api-access-m45qt\") pod \"openshift-state-metrics-7bd7489957-5vx6k\" (UID: \"cd430b13-1a61-4f19-8139-791ac71c1eff\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.770351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770317 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-root\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770395 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-sys\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770433 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-wtmp\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770459 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hv2\" (UniqueName: \"kubernetes.io/projected/3ec9b82f-9d14-4035-a558-d7014e495f23-kube-api-access-p7hv2\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770490 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770581 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec9b82f-9d14-4035-a558-d7014e495f23-metrics-client-ca\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770626 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-accelerators-collector-config\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770688 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-tls\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.770764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.770721 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-textfile\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.792568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.792544 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 12:22:47.792675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.792659 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 12:22:47.792740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.792726 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 12:22:47.792773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.792742 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ttg6g\"" Nov 01 12:22:47.847641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.847615 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" Nov 01 12:22:47.871833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.871798 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec9b82f-9d14-4035-a558-d7014e495f23-metrics-client-ca\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.871933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.871855 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-accelerators-collector-config\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.871933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.871894 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-tls\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.871933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.871928 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-textfile\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.872050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.871967 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-root\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.872050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.871997 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-sys\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.872050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.872019 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-wtmp\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.872050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.872043 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hv2\" (UniqueName: \"kubernetes.io/projected/3ec9b82f-9d14-4035-a558-d7014e495f23-kube-api-access-p7hv2\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.872179 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.872076 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.874009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.873006 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-textfile\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.874009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.873192 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-sys\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.874009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.873368 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-wtmp\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.874009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.873565 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-accelerators-collector-config\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.874009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.873622 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec9b82f-9d14-4035-a558-d7014e495f23-root\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.874009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.873968 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec9b82f-9d14-4035-a558-d7014e495f23-metrics-client-ca\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.875978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.875944 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.877189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.877165 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec9b82f-9d14-4035-a558-d7014e495f23-node-exporter-tls\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.917143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.917049 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:47.925564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.925542 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hv2\" (UniqueName: \"kubernetes.io/projected/3ec9b82f-9d14-4035-a558-d7014e495f23-kube-api-access-p7hv2\") pod \"node-exporter-l76dc\" (UID: \"3ec9b82f-9d14-4035-a558-d7014e495f23\") " pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:47.995204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:47.995176 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l76dc" Nov 01 12:22:48.007074 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:48.007048 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec9b82f_9d14_4035_a558_d7014e495f23.slice/crio-15d442ff7bd06ed38a92765c013f0b11d576409365d19690a0d61333da4d52e2 WatchSource:0}: Error finding container 15d442ff7bd06ed38a92765c013f0b11d576409365d19690a0d61333da4d52e2: Status 404 returned error can't find the container with id 15d442ff7bd06ed38a92765c013f0b11d576409365d19690a0d61333da4d52e2 Nov 01 12:22:48.057409 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.057377 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:48.259138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.259101 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k"] Nov 01 12:22:48.262266 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:48.262239 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd430b13_1a61_4f19_8139_791ac71c1eff.slice/crio-f6efa52e6987444d41573ddb2336a52ef5dab7ee5e0fcb88a8840af30a160ec4 WatchSource:0}: Error finding container f6efa52e6987444d41573ddb2336a52ef5dab7ee5e0fcb88a8840af30a160ec4: Status 404 returned error can't find the container with id f6efa52e6987444d41573ddb2336a52ef5dab7ee5e0fcb88a8840af30a160ec4 Nov 01 12:22:48.525680 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.525597 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l76dc" event={"ID":"3ec9b82f-9d14-4035-a558-d7014e495f23","Type":"ContainerStarted","Data":"15d442ff7bd06ed38a92765c013f0b11d576409365d19690a0d61333da4d52e2"} Nov 01 12:22:48.527357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.527334 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" event={"ID":"cd430b13-1a61-4f19-8139-791ac71c1eff","Type":"ContainerStarted","Data":"34e00b2abd46f436b5e6e6ae20709a5189c9727702a38880ced9a0e87c4fb781"} Nov 01 12:22:48.527453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.527381 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" event={"ID":"cd430b13-1a61-4f19-8139-791ac71c1eff","Type":"ContainerStarted","Data":"796f187b98e9ef330ee6e57dd9dc2c4d5a3b31176522ebdc8e903c91ff278940"} Nov 01 12:22:48.527453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.527401 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" event={"ID":"cd430b13-1a61-4f19-8139-791ac71c1eff","Type":"ContainerStarted","Data":"f6efa52e6987444d41573ddb2336a52ef5dab7ee5e0fcb88a8840af30a160ec4"} Nov 01 12:22:48.902399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:48.902315 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:49.115068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:49.115031 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:49.739711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:49.739669 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:50.535805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:50.535773 4484 generic.go:356] "Generic (PLEG): container finished" podID="3ec9b82f-9d14-4035-a558-d7014e495f23" containerID="f207d3687ae7fa8d5d4c83a1569841f271c0c252b342b4e17e3a660eee513edd" exitCode=0 Nov 01 12:22:50.535960 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:50.535851 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l76dc" event={"ID":"3ec9b82f-9d14-4035-a558-d7014e495f23","Type":"ContainerDied","Data":"f207d3687ae7fa8d5d4c83a1569841f271c0c252b342b4e17e3a660eee513edd"} Nov 01 12:22:50.538042 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:50.538017 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" event={"ID":"cd430b13-1a61-4f19-8139-791ac71c1eff","Type":"ContainerStarted","Data":"0864e9cb5369154f36e29bf9280d5c3e53d896f629d3d70a75d89662b3161a40"} Nov 01 12:22:50.795575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:50.795454 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5vx6k" podStartSLOduration=2.062798421 podStartE2EDuration="3.795432311s" totalImagesPullingTime="1.73263389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:50.79437441 +0000 UTC m=+40.544317387" watchObservedRunningTime="2026-11-01 12:22:50.795432311 +0000 UTC m=+40.545375291" Nov 01 12:22:51.472027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.471993 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bc77dddf-nsdgr" Nov 01 12:22:51.538165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.538136 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:51.542613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.542587 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l76dc" event={"ID":"3ec9b82f-9d14-4035-a558-d7014e495f23","Type":"ContainerStarted","Data":"02cf5381e4113c4f38a0a05028da4931952a90c09e40c4d006df2a57dcdb56cf"} Nov 01 12:22:51.542758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.542634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l76dc" event={"ID":"3ec9b82f-9d14-4035-a558-d7014e495f23","Type":"ContainerStarted","Data":"0d0483b8c7488e9eb65f96e9855f87d31c62751cb9d8cce2bbb0517358472b36"} Nov 01 12:22:51.657287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.657256 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:51.690398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.690349 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l76dc" podStartSLOduration=3.06848956 podStartE2EDuration="4.690335426s" totalImagesPullingTime="1.552223582s" totalInitContainerRuntime="69.622284ms" isStatefulPod=true podCreationTimestamp="2026-11-01 12:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:51.689916744 +0000 UTC m=+41.439859700" watchObservedRunningTime="2026-11-01 12:22:51.690335426 +0000 UTC m=+41.440278400" Nov 01 12:22:51.773857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.773818 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:51.884430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:51.884397 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:52.180411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.180320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:52.289987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.289950 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:52.357547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.357508 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s"] Nov 01 12:22:52.361020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.360999 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.403501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.403463 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:52.408567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.408544 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4qs\" (UniqueName: \"kubernetes.io/projected/ab722746-95b1-45a9-a6f3-efce8f887804-kube-api-access-ph4qs\") pod \"monitoring-plugin-77f4f76594-jvq5s\" (UID: \"ab722746-95b1-45a9-a6f3-efce8f887804\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.408646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.408578 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab722746-95b1-45a9-a6f3-efce8f887804-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jvq5s\" (UID: \"ab722746-95b1-45a9-a6f3-efce8f887804\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.409352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.409338 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jv6t6\"" Nov 01 12:22:52.455255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.455187 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 12:22:52.455936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.455914 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s"] Nov 01 12:22:52.509286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.509255 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4qs\" (UniqueName: \"kubernetes.io/projected/ab722746-95b1-45a9-a6f3-efce8f887804-kube-api-access-ph4qs\") pod \"monitoring-plugin-77f4f76594-jvq5s\" (UID: \"ab722746-95b1-45a9-a6f3-efce8f887804\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.509468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.509294 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab722746-95b1-45a9-a6f3-efce8f887804-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jvq5s\" (UID: \"ab722746-95b1-45a9-a6f3-efce8f887804\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.511728 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.511702 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab722746-95b1-45a9-a6f3-efce8f887804-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jvq5s\" (UID: \"ab722746-95b1-45a9-a6f3-efce8f887804\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.561513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.561482 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4qs\" (UniqueName: \"kubernetes.io/projected/ab722746-95b1-45a9-a6f3-efce8f887804-kube-api-access-ph4qs\") pod \"monitoring-plugin-77f4f76594-jvq5s\" (UID: \"ab722746-95b1-45a9-a6f3-efce8f887804\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:52.675062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:52.675014 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:53.032318 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:53.032272 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s"] Nov 01 12:22:53.036068 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:22:53.036042 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab722746_95b1_45a9_a6f3_efce8f887804.slice/crio-d092dcee0aa9b36e2dfbc82f3aa81a65460ae5911f170909c9c7a3fce6aaccdb WatchSource:0}: Error finding container d092dcee0aa9b36e2dfbc82f3aa81a65460ae5911f170909c9c7a3fce6aaccdb: Status 404 returned error can't find the container with id d092dcee0aa9b36e2dfbc82f3aa81a65460ae5911f170909c9c7a3fce6aaccdb Nov 01 12:22:53.550072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:53.549995 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" event={"ID":"ab722746-95b1-45a9-a6f3-efce8f887804","Type":"ContainerStarted","Data":"d092dcee0aa9b36e2dfbc82f3aa81a65460ae5911f170909c9c7a3fce6aaccdb"} Nov 01 12:22:53.937204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:53.937130 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:54.050132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:54.050099 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:54.174224 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:54.174196 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:54.286716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:54.286665 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:54.398743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:54.398708 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:55.559730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:55.559699 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" event={"ID":"ab722746-95b1-45a9-a6f3-efce8f887804","Type":"ContainerStarted","Data":"bb131bf047ed42996fe3e8c21fc9c024d5acf4c61d34505af8f48f7f96655918"} Nov 01 12:22:55.560196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:55.559751 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:55.565030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:55.565010 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" Nov 01 12:22:55.670901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:55.670847 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jvq5s" podStartSLOduration=1.6728843009999999 podStartE2EDuration="3.670828471s" totalImagesPullingTime="1.99794417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:55.669778157 +0000 UTC m=+45.419721135" watchObservedRunningTime="2026-11-01 12:22:55.670828471 +0000 UTC m=+45.420771452" Nov 01 12:22:56.415128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:56.415091 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:58.613210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:58.613172 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:58.722729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:58.722698 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:58.831968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:58.831938 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:59.279756 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:59.279726 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:59.396484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:59.396454 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:59.801831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:59.801802 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:22:59.940493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:22:59.940462 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:00.251335 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:23:00.251288 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:23:01.131932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.131898 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:01.243111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.243076 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:01.409295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.409224 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-r5bbx-8fck5"] Nov 01 12:23:01.409546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.409524 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" podUID="129138ad-2f7a-4994-8693-61c7b660f53f" containerName="insights-gathering" containerID="cri-o://39a4181d8cf857e017284f9573a659a99ffc06338fe49b07c71edc7fee844281" gracePeriod=30 Nov 01 12:23:01.579693 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.579661 4484 generic.go:356] "Generic (PLEG): container finished" podID="129138ad-2f7a-4994-8693-61c7b660f53f" containerID="39a4181d8cf857e017284f9573a659a99ffc06338fe49b07c71edc7fee844281" exitCode=2 Nov 01 12:23:01.579817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.579744 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" event={"ID":"129138ad-2f7a-4994-8693-61c7b660f53f","Type":"ContainerDied","Data":"39a4181d8cf857e017284f9573a659a99ffc06338fe49b07c71edc7fee844281"} Nov 01 12:23:01.641291 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.641264 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:23:01.774523 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774482 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") pod \"129138ad-2f7a-4994-8693-61c7b660f53f\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " Nov 01 12:23:01.774715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") pod \"129138ad-2f7a-4994-8693-61c7b660f53f\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " Nov 01 12:23:01.774715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774589 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") pod \"129138ad-2f7a-4994-8693-61c7b660f53f\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " Nov 01 12:23:01.774715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774622 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") pod \"129138ad-2f7a-4994-8693-61c7b660f53f\" (UID: \"129138ad-2f7a-4994-8693-61c7b660f53f\") " Nov 01 12:23:01.774919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774893 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path" pod "129138ad-2f7a-4994-8693-61c7b660f53f" (UID: "129138ad-2f7a-4994-8693-61c7b660f53f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:23:01.774958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774912 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle" pod "129138ad-2f7a-4994-8693-61c7b660f53f" (UID: "129138ad-2f7a-4994-8693-61c7b660f53f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:23:01.775008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.774985 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle" pod "129138ad-2f7a-4994-8693-61c7b660f53f" (UID: "129138ad-2f7a-4994-8693-61c7b660f53f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:23:01.776895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.776876 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg" pod "129138ad-2f7a-4994-8693-61c7b660f53f" (UID: "129138ad-2f7a-4994-8693-61c7b660f53f"). InnerVolumeSpecName "kube-api-access-xgltg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:23:01.875549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.875496 4484 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-trusted-ca-bundle\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:23:01.875549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.875540 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgltg\" (UniqueName: \"kubernetes.io/projected/129138ad-2f7a-4994-8693-61c7b660f53f-kube-api-access-xgltg\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:23:01.875549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.875553 4484 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/129138ad-2f7a-4994-8693-61c7b660f53f-archives-path\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:23:01.875549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:01.875563 4484 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129138ad-2f7a-4994-8693-61c7b660f53f-service-ca-bundle\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:23:02.583974 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:02.583943 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" Nov 01 12:23:02.584427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:02.583940 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-r5bbx-8fck5" event={"ID":"129138ad-2f7a-4994-8693-61c7b660f53f","Type":"ContainerDied","Data":"9b7711bdb742154e856a58150dd5d16ad11176cc5f5b307a4d53774b55ff955d"} Nov 01 12:23:02.584427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:02.584085 4484 scope.go:122] "RemoveContainer" containerID="39a4181d8cf857e017284f9573a659a99ffc06338fe49b07c71edc7fee844281" Nov 01 12:23:02.733794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:02.733764 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-r5bbx-8fck5"] Nov 01 12:23:02.779942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:02.779912 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-r5bbx-8fck5"] Nov 01 12:23:04.366336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:04.366283 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129138ad-2f7a-4994-8693-61c7b660f53f" path="/var/lib/kubelet/pods/129138ad-2f7a-4994-8693-61c7b660f53f/volumes" Nov 01 12:23:06.472206 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.472172 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d544bdc6-psxd4"] Nov 01 12:23:06.472611 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.472456 4484 cpu_manager.go:414] "containerMap: removing container" podUID="129138ad-2f7a-4994-8693-61c7b660f53f" containerName="insights-gathering" Nov 01 12:23:06.472611 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.472469 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129138ad-2f7a-4994-8693-61c7b660f53f" containerName="insights-gathering" Nov 01 12:23:06.472611 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.472528 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="129138ad-2f7a-4994-8693-61c7b660f53f" containerName="insights-gathering" Nov 01 12:23:06.474546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.474529 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.541916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.541885 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 12:23:06.542072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.541930 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 12:23:06.569370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.569334 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 12:23:06.569525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.569398 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 12:23:06.569525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.569467 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 12:23:06.607427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607381 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607425 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607451 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj687\" (UniqueName: \"kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607530 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607560 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607587 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607676 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.607744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.607723 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.662648 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.662612 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 12:23:06.663641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.663606 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d544bdc6-psxd4"] Nov 01 12:23:06.680699 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.680674 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-69snc\"" Nov 01 12:23:06.680699 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.680679 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 12:23:06.680865 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.680730 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 12:23:06.708051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708017 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708054 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708078 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj687\" (UniqueName: \"kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708094 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708118 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708149 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708181 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708230 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708599 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.708879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.709022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.708999 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.709239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.709216 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.709340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.709320 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.710672 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.710653 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.710745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.710718 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.736443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.736415 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:06.761024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.761001 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj687\" (UniqueName: \"kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687\") pod \"console-5d544bdc6-psxd4\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.784944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.784915 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:06.850614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:06.850039 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:07.192251 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:07.192155 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d544bdc6-psxd4"] Nov 01 12:23:07.196118 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:23:07.196074 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5834bf8f_5873_40aa_a829_44ccbb4111a7.slice/crio-4adbf7fa0e699378795276edec560095cabaca6771ec5882fcc9e74d2508fecb WatchSource:0}: Error finding container 4adbf7fa0e699378795276edec560095cabaca6771ec5882fcc9e74d2508fecb: Status 404 returned error can't find the container with id 4adbf7fa0e699378795276edec560095cabaca6771ec5882fcc9e74d2508fecb Nov 01 12:23:07.600136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:07.600101 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d544bdc6-psxd4" event={"ID":"5834bf8f-5873-40aa-a829-44ccbb4111a7","Type":"ContainerStarted","Data":"4adbf7fa0e699378795276edec560095cabaca6771ec5882fcc9e74d2508fecb"} Nov 01 12:23:11.156565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:11.156522 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:11.275594 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:11.275561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:11.617281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:11.617246 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d544bdc6-psxd4" event={"ID":"5834bf8f-5873-40aa-a829-44ccbb4111a7","Type":"ContainerStarted","Data":"cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf"} Nov 01 12:23:11.724456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:11.724410 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d544bdc6-psxd4" podStartSLOduration=2.370550852 podStartE2EDuration="5.724396618s" totalImagesPullingTime="3.353845766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:11.723199206 +0000 UTC m=+61.473142196" watchObservedRunningTime="2026-11-01 12:23:11.724396618 +0000 UTC m=+61.474339595" Nov 01 12:23:14.310520 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:14.310470 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:14.423550 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:14.423518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:15.935332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:15.935286 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:16.063364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:16.063331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:16.785487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:16.785456 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:16.785659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:16.785527 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:16.791267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:16.791246 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:17.638514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:17.638484 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:23:24.095430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:24.095393 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:26.340849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:26.340813 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:26.450186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:26.450147 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:29.279261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:29.279213 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:29.389288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:29.389213 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:29.943370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:29.943328 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:30.054205 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:30.054161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:30.250047 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:23:30.250014 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:23:38.927875 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:38.927847 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:39.229559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:39.229522 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:44.282347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:44.282317 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:44.392013 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:44.391979 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:50.197567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.197524 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-87877c8d-7q97w"] Nov 01 12:23:50.205286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.205265 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220502 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220536 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220554 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5655\" (UniqueName: \"kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220663 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220710 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220739 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220753 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.220833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.220770 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.256952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.256920 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87877c8d-7q97w"] Nov 01 12:23:50.321253 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321225 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321260 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321277 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5655\" (UniqueName: \"kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321311 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321336 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321371 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321385 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.321898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321868 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.322138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.322112 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.322220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.322169 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.322220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.322168 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.322220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.321409 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.322420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.322385 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.323827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.323803 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.323915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.323886 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.375590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.375564 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5655\" (UniqueName: \"kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655\") pod \"console-87877c8d-7q97w\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.452400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.452291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:50.515921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.515888 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:23:50.565155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.565084 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:50.846515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.846473 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:50.875559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.875529 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87877c8d-7q97w"] Nov 01 12:23:50.879058 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:23:50.879029 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e38fe5_dfaa_4316_85d4_79e26ad5e189.slice/crio-2e3beda45ecf5168c7d11a170b68b50489c506063d9908bae172c71059665803 WatchSource:0}: Error finding container 2e3beda45ecf5168c7d11a170b68b50489c506063d9908bae172c71059665803: Status 404 returned error can't find the container with id 2e3beda45ecf5168c7d11a170b68b50489c506063d9908bae172c71059665803 Nov 01 12:23:50.956465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:50.956430 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:23:51.739946 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:51.739910 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87877c8d-7q97w" event={"ID":"54e38fe5-dfaa-4316-85d4-79e26ad5e189","Type":"ContainerStarted","Data":"7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91"} Nov 01 12:23:51.740344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:51.739962 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87877c8d-7q97w" event={"ID":"54e38fe5-dfaa-4316-85d4-79e26ad5e189","Type":"ContainerStarted","Data":"2e3beda45ecf5168c7d11a170b68b50489c506063d9908bae172c71059665803"} Nov 01 12:23:51.846311 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:23:51.846249 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-87877c8d-7q97w" podStartSLOduration=1.8462365059999999 podStartE2EDuration="1.846236506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:51.846129709 +0000 UTC m=+101.596072687" watchObservedRunningTime="2026-11-01 12:23:51.846236506 +0000 UTC m=+101.596179483" Nov 01 12:24:00.137111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.137074 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:24:00.243793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.243764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:24:00.250344 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:24:00.250315 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:24:00.516482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.516457 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:24:00.516609 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.516510 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:24:00.521094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.521068 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:24:00.777024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.776947 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:24:00.913627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:00.913596 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d544bdc6-psxd4"] Nov 01 12:24:25.932701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:25.932597 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d544bdc6-psxd4" podUID="5834bf8f-5873-40aa-a829-44ccbb4111a7" containerName="console" containerID="cri-o://cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf" gracePeriod=15 Nov 01 12:24:26.187688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.187627 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:24:26.260095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260063 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260112 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260165 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260191 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260227 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260257 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260322 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260353 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687\" (UniqueName: \"kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687\") pod \"5834bf8f-5873-40aa-a829-44ccbb4111a7\" (UID: \"5834bf8f-5873-40aa-a829-44ccbb4111a7\") " Nov 01 12:24:26.260695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260556 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:24:26.260695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260606 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:24:26.260876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260753 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:24:26.260876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260679 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:24:26.260876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.260798 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:24:26.262490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.262466 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:24:26.262575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.262549 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "kube-api-access-fj687". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:24:26.262633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.262611 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert" pod "5834bf8f-5873-40aa-a829-44ccbb4111a7" (UID: "5834bf8f-5873-40aa-a829-44ccbb4111a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:24:26.361006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.360976 4484 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-trusted-ca-bundle\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361000 4484 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-serving-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361010 4484 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-config\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361019 4484 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-oauth-serving-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361028 4484 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5834bf8f-5873-40aa-a829-44ccbb4111a7-tmp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361037 4484 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5834bf8f-5873-40aa-a829-44ccbb4111a7-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361045 4484 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5834bf8f-5873-40aa-a829-44ccbb4111a7-console-oauth-config\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.361210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.361055 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj687\" (UniqueName: \"kubernetes.io/projected/5834bf8f-5873-40aa-a829-44ccbb4111a7-kube-api-access-fj687\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:24:26.853217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.853179 4484 generic.go:356] "Generic (PLEG): container finished" podID="5834bf8f-5873-40aa-a829-44ccbb4111a7" containerID="cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf" exitCode=0 Nov 01 12:24:26.853404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.853232 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d544bdc6-psxd4" Nov 01 12:24:26.853404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.853266 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d544bdc6-psxd4" event={"ID":"5834bf8f-5873-40aa-a829-44ccbb4111a7","Type":"ContainerDied","Data":"cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf"} Nov 01 12:24:26.853404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.853336 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d544bdc6-psxd4" event={"ID":"5834bf8f-5873-40aa-a829-44ccbb4111a7","Type":"ContainerDied","Data":"4adbf7fa0e699378795276edec560095cabaca6771ec5882fcc9e74d2508fecb"} Nov 01 12:24:26.853404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.853358 4484 scope.go:122] "RemoveContainer" containerID="cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf" Nov 01 12:24:26.860745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.860729 4484 scope.go:122] "RemoveContainer" containerID="cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf" Nov 01 12:24:26.861005 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:24:26.860979 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf\": container with ID starting with cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf not found: ID does not exist" containerID="cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf" Nov 01 12:24:26.861052 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:26.861010 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf"} err="failed to get container status \"cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf\": rpc error: code = NotFound desc = could not find container \"cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf\": container with ID starting with cf964ebb42be6a4f7a8734ddb10c05354c1fc6074bb42a373270cd7b6a5968bf not found: ID does not exist" Nov 01 12:24:27.004413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:27.004389 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d544bdc6-psxd4"] Nov 01 12:24:27.050400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:27.050379 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d544bdc6-psxd4"] Nov 01 12:24:28.365794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:28.365765 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5834bf8f-5873-40aa-a829-44ccbb4111a7" path="/var/lib/kubelet/pods/5834bf8f-5873-40aa-a829-44ccbb4111a7/volumes" Nov 01 12:24:30.249950 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:24:30.249913 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:24:56.631873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:56.631835 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:24:56.739932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:24:56.739898 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:00.249568 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:25:00.249528 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:25:00.460834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:00.460796 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:09.297735 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:09.297703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:09.405533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:09.405501 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:12.959564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:12.959522 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c75b94f74-6jxw6"] Nov 01 12:25:12.960068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:12.959891 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5834bf8f-5873-40aa-a829-44ccbb4111a7" containerName="console" Nov 01 12:25:12.960068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:12.959909 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5834bf8f-5873-40aa-a829-44ccbb4111a7" containerName="console" Nov 01 12:25:12.960068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:12.959972 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5834bf8f-5873-40aa-a829-44ccbb4111a7" containerName="console" Nov 01 12:25:12.964027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:12.964004 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.056715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.056681 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c75b94f74-6jxw6"] Nov 01 12:25:13.084732 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.084696 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-serving-cert\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.084732 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.084730 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-tmp\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.084958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.084749 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-config\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.084958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.084771 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-oauth-config\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.084958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.084844 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-oauth-serving-cert\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.085113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.085014 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-trusted-ca-bundle\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.085113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.085046 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-service-ca\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.085181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.085163 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqg8\" (UniqueName: \"kubernetes.io/projected/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-kube-api-access-2sqg8\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186535 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqg8\" (UniqueName: \"kubernetes.io/projected/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-kube-api-access-2sqg8\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186578 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-serving-cert\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186599 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-tmp\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186626 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-config\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186661 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-oauth-config\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186683 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-oauth-serving-cert\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186711 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-trusted-ca-bundle\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.186930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.186739 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-service-ca\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.187037 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.187016 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-tmp\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.187506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.187483 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-config\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.187611 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.187592 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-trusted-ca-bundle\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.187655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.187615 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-oauth-serving-cert\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.187655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.187625 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-service-ca\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.189662 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.189640 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-serving-cert\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.189751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.189685 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-console-oauth-config\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.216246 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.216181 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:13.238898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.238875 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqg8\" (UniqueName: \"kubernetes.io/projected/4b1e87e4-fa6b-4ea4-bc30-485d714be38d-kube-api-access-2sqg8\") pod \"console-6c75b94f74-6jxw6\" (UID: \"4b1e87e4-fa6b-4ea4-bc30-485d714be38d\") " pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.275294 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.275267 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:13.330858 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.330824 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:13.611850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.611816 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:13.684241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.684210 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c75b94f74-6jxw6"] Nov 01 12:25:13.688102 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:25:13.688069 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1e87e4_fa6b_4ea4_bc30_485d714be38d.slice/crio-70790fa33b9a7d10d41c5be8c015673adddb04f044cd40b7b9048226fcec37af WatchSource:0}: Error finding container 70790fa33b9a7d10d41c5be8c015673adddb04f044cd40b7b9048226fcec37af: Status 404 returned error can't find the container with id 70790fa33b9a7d10d41c5be8c015673adddb04f044cd40b7b9048226fcec37af Nov 01 12:25:13.724852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.724826 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:13.998621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.998586 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c75b94f74-6jxw6" event={"ID":"4b1e87e4-fa6b-4ea4-bc30-485d714be38d","Type":"ContainerStarted","Data":"a20dafd4ac46750ae2bae427f77b27b969feefb6f1fa82c096d5433fd5cced44"} Nov 01 12:25:13.999019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:13.998641 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c75b94f74-6jxw6" event={"ID":"4b1e87e4-fa6b-4ea4-bc30-485d714be38d","Type":"ContainerStarted","Data":"70790fa33b9a7d10d41c5be8c015673adddb04f044cd40b7b9048226fcec37af"} Nov 01 12:25:14.106591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:14.106533 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c75b94f74-6jxw6" podStartSLOduration=2.106516195 podStartE2EDuration="2.106516195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:25:14.105166092 +0000 UTC m=+183.855109070" watchObservedRunningTime="2026-11-01 12:25:14.106516195 +0000 UTC m=+183.856459174" Nov 01 12:25:14.285133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:14.285021 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:14.403284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:14.403249 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:25:23.276019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:23.275975 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:23.276532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:23.276046 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:23.280875 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:23.280851 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:24.031571 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:24.031543 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c75b94f74-6jxw6" Nov 01 12:25:24.184058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:24.184026 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87877c8d-7q97w"] Nov 01 12:25:30.249850 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:25:30.249814 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:25:49.203347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.203269 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-87877c8d-7q97w" podUID="54e38fe5-dfaa-4316-85d4-79e26ad5e189" containerName="console" containerID="cri-o://7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91" gracePeriod=15 Nov 01 12:25:49.439785 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.439760 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:25:49.551463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551431 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.551650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551484 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.551650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551518 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.551650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551548 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655\" (UniqueName: \"kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.551650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551591 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.551650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551623 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551665 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.551706 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp\") pod \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\" (UID: \"54e38fe5-dfaa-4316-85d4-79e26ad5e189\") " Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.552116 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.552333 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.552341 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.552472 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:25:49.553272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.553140 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:25:49.554146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.554117 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:25:49.554441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.554420 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "kube-api-access-l5655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:25:49.554441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.554430 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert" pod "54e38fe5-dfaa-4316-85d4-79e26ad5e189" (UID: "54e38fe5-dfaa-4316-85d4-79e26ad5e189"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:25:49.652617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652580 4484 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-trusted-ca-bundle\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652610 4484 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e38fe5-dfaa-4316-85d4-79e26ad5e189-tmp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652620 4484 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-oauth-config\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652630 4484 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-config\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652639 4484 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-oauth-serving-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652649 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5655\" (UniqueName: \"kubernetes.io/projected/54e38fe5-dfaa-4316-85d4-79e26ad5e189-kube-api-access-l5655\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652661 4484 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54e38fe5-dfaa-4316-85d4-79e26ad5e189-console-serving-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:49.652834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:49.652669 4484 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54e38fe5-dfaa-4316-85d4-79e26ad5e189-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:25:50.115466 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.115433 4484 generic.go:356] "Generic (PLEG): container finished" podID="54e38fe5-dfaa-4316-85d4-79e26ad5e189" containerID="7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91" exitCode=0 Nov 01 12:25:50.115614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.115481 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87877c8d-7q97w" Nov 01 12:25:50.115614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.115503 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87877c8d-7q97w" event={"ID":"54e38fe5-dfaa-4316-85d4-79e26ad5e189","Type":"ContainerDied","Data":"7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91"} Nov 01 12:25:50.115614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.115571 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87877c8d-7q97w" event={"ID":"54e38fe5-dfaa-4316-85d4-79e26ad5e189","Type":"ContainerDied","Data":"2e3beda45ecf5168c7d11a170b68b50489c506063d9908bae172c71059665803"} Nov 01 12:25:50.115614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.115591 4484 scope.go:122] "RemoveContainer" containerID="7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91" Nov 01 12:25:50.124032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.124012 4484 scope.go:122] "RemoveContainer" containerID="7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91" Nov 01 12:25:50.124286 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:25:50.124269 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91\": container with ID starting with 7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91 not found: ID does not exist" containerID="7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91" Nov 01 12:25:50.124368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.124294 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91"} err="failed to get container status \"7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91\": rpc error: code = NotFound desc = could not find container \"7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91\": container with ID starting with 7332ddb2ff736493eabeaeb927a981c015e6477f9c2b609a88cb95a6ac1f5d91 not found: ID does not exist" Nov 01 12:25:50.265800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.265747 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-87877c8d-7q97w"] Nov 01 12:25:50.311856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.311831 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-87877c8d-7q97w"] Nov 01 12:25:50.366181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:25:50.366121 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e38fe5-dfaa-4316-85d4-79e26ad5e189" path="/var/lib/kubelet/pods/54e38fe5-dfaa-4316-85d4-79e26ad5e189/volumes" Nov 01 12:26:00.249356 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:26:00.249264 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:26:00.803154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:26:00.803106 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:26:22.735693 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:26:22.735659 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:26:22.846238 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:26:22.846207 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:26:30.250195 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:26:30.250151 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:27:00.250334 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:27:00.250281 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:27:01.146984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:01.146936 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:02.298239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:02.298165 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:18.069420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:18.069388 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:18.233712 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:18.233675 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:19.347019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.346991 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2"] Nov 01 12:27:19.347401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.347263 4484 cpu_manager.go:414] "containerMap: removing container" podUID="54e38fe5-dfaa-4316-85d4-79e26ad5e189" containerName="console" Nov 01 12:27:19.347401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.347275 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e38fe5-dfaa-4316-85d4-79e26ad5e189" containerName="console" Nov 01 12:27:19.347401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.347350 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e38fe5-dfaa-4316-85d4-79e26ad5e189" containerName="console" Nov 01 12:27:19.350111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.350094 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359382 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359419 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359526 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359556 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359652 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359669 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lr7\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359760 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359717 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.359982 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.359967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:19.399703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399645 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 12:27:19.399703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399645 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 12:27:19.399703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399671 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 12:27:19.399703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399698 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 12:27:19.399941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399733 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 12:27:19.399941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399778 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 12:27:19.399941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.399780 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mmr7k\"" Nov 01 12:27:19.448201 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.448174 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2"] Nov 01 12:27:19.460958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.460933 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.460971 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.460994 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.461047 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.461075 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.461108 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.461133 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lr7\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.461793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.461769 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.463466 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.463421 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.463554 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.463469 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.463721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.463702 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.463864 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.463847 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.481423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.481396 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:19.513774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.513739 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.513957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.513938 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lr7\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7\") pod \"istiod-openshift-gateway-7876bfcb74-lnkf2\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:19.661425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:19.661344 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:20.072834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:20.072799 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2"] Nov 01 12:27:20.075789 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:27:20.075756 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7359aaa2_ae08_460a_bc56_9fb7e1588fc9.slice/crio-8e395a0ba6c8460f13b2fb1eb16a28e20b95f870f327e27364ed63855ca6a5a5 WatchSource:0}: Error finding container 8e395a0ba6c8460f13b2fb1eb16a28e20b95f870f327e27364ed63855ca6a5a5: Status 404 returned error can't find the container with id 8e395a0ba6c8460f13b2fb1eb16a28e20b95f870f327e27364ed63855ca6a5a5 Nov 01 12:27:20.077689 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:20.077673 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:27:20.317254 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:20.317217 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:20.403964 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:20.403889 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" event={"ID":"7359aaa2-ae08-460a-bc56-9fb7e1588fc9","Type":"ContainerStarted","Data":"8e395a0ba6c8460f13b2fb1eb16a28e20b95f870f327e27364ed63855ca6a5a5"} Nov 01 12:27:23.948917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:23.948886 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:27:23.949175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:23.948948 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:27:24.418834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:24.418798 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" event={"ID":"7359aaa2-ae08-460a-bc56-9fb7e1588fc9","Type":"ContainerStarted","Data":"6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd"} Nov 01 12:27:24.418996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:24.418956 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:24.420516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:24.420490 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lnkf2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:27:24.420637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:24.420539 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:27:24.609949 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:24.609890 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" podStartSLOduration=1.7395236170000001 podStartE2EDuration="5.609876369s" totalImagesPullingTime="3.870352752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:27:24.572714838 +0000 UTC m=+314.322657816" watchObservedRunningTime="2026-11-01 12:27:24.609876369 +0000 UTC m=+314.359819347" Nov 01 12:27:25.177609 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:25.177574 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:25.310733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:25.310690 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:25.422604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:25.422579 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lnkf2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:27:25.422737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:25.422630 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:27:26.115217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:26.115183 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:26.409453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:26.409421 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:26.524384 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:26.524332 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:28.423751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:28.423725 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:27:29.993197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:29.993165 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:30.250392 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:27:30.250358 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:27:41.954936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:41.954902 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:27:42.067284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:27:42.067249 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:00.250568 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:28:00.250532 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:28:01.498075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:01.498017 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:04.861678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:04.861638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:04.968383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:04.968350 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:07.811488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:07.811444 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:07.926860 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:07.926824 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:17.489895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:17.489864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:17.653386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:17.653353 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:26.226010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:26.225974 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:27.219738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:27.219703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:29.040250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:29.040211 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:29.210744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:29.210704 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:30.250266 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:28:30.250224 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:28:30.425322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.425278 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:30.632086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.632017 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v"] Nov 01 12:28:30.635273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.635253 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.668920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.668895 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:30.687552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.687533 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 12:28:30.687673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.687584 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 12:28:30.687673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.687624 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 12:28:30.687771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.687738 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 12:28:30.687940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.687924 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hqxjc\"" Nov 01 12:28:30.733551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.733528 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 12:28:30.784159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.784135 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.784267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.784172 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7kc\" (UniqueName: \"kubernetes.io/projected/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-kube-api-access-tv7kc\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.784267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.784201 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.784359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.784272 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.784392 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.784353 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.784425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.784393 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.793638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.793616 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:30.885147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885094 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.885147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885123 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.885290 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885146 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.885290 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885171 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.885290 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885204 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.885290 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885229 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7kc\" (UniqueName: \"kubernetes.io/projected/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-kube-api-access-tv7kc\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.885621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.885593 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.887427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.887401 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.887644 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.887628 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.887698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.887682 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.887851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.887829 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.906487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.906461 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v"] Nov 01 12:28:30.917706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.917685 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:30.938309 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.938276 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7kc\" (UniqueName: \"kubernetes.io/projected/6b49da42-807d-4f9d-9fdf-6b2556cbe25e-kube-api-access-tv7kc\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-rq24v\" (UID: \"6b49da42-807d-4f9d-9fdf-6b2556cbe25e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:30.950117 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:30.950098 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:31.460921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:31.460885 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v"] Nov 01 12:28:31.464412 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:28:31.464377 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b49da42_807d_4f9d_9fdf_6b2556cbe25e.slice/crio-ee3bda97600f61c94a8e832f15afeba6db342d2a7bf83f49d0afc66bbd880674 WatchSource:0}: Error finding container ee3bda97600f61c94a8e832f15afeba6db342d2a7bf83f49d0afc66bbd880674: Status 404 returned error can't find the container with id ee3bda97600f61c94a8e832f15afeba6db342d2a7bf83f49d0afc66bbd880674 Nov 01 12:28:31.638675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:31.638639 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" event={"ID":"6b49da42-807d-4f9d-9fdf-6b2556cbe25e","Type":"ContainerStarted","Data":"ee3bda97600f61c94a8e832f15afeba6db342d2a7bf83f49d0afc66bbd880674"} Nov 01 12:28:33.033457 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:33.033416 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:34.851413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:34.851380 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:35.784145 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:35.784093 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:36.260051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:36.260014 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:36.831452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:36.831427 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:37.663418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:37.663344 4484 generic.go:356] "Generic (PLEG): container finished" podID="6b49da42-807d-4f9d-9fdf-6b2556cbe25e" containerID="2b95dd3e8d817ec4515e8ea5797710304a43821b1ff3cb095d5d05f6621d1c28" exitCode=0 Nov 01 12:28:37.663768 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:37.663428 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" event={"ID":"6b49da42-807d-4f9d-9fdf-6b2556cbe25e","Type":"ContainerDied","Data":"2b95dd3e8d817ec4515e8ea5797710304a43821b1ff3cb095d5d05f6621d1c28"} Nov 01 12:28:38.633730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:38.633701 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:38.670108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:38.670075 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" event={"ID":"6b49da42-807d-4f9d-9fdf-6b2556cbe25e","Type":"ContainerStarted","Data":"ba7d02133cf9e01417e4f68d1ba0e5c46edeb1d0a519018c0ba6f36a6b9bc912"} Nov 01 12:28:38.670523 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:38.670216 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:38.887218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:38.887114 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" podStartSLOduration=2.208715727 podStartE2EDuration="8.88666832s" totalImagesPullingTime="6.190396551s" totalInitContainerRuntime="487.556042ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:38.83908672 +0000 UTC m=+388.589029708" watchObservedRunningTime="2026-11-01 12:28:38.88666832 +0000 UTC m=+388.636611301" Nov 01 12:28:39.074386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:39.074347 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:40.835098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:40.835061 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:41.472416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:41.472388 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:41.629511 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:41.629479 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:42.639901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:42.639872 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:43.301085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:43.301049 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:43.448372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:43.448344 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:44.398340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:44.398289 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:49.105466 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:49.105433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:49.243489 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:49.243449 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:49.675903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:49.675876 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-rq24v" Nov 01 12:28:50.048403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:50.048370 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:50.173471 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:50.173444 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:51.113347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:51.113316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:51.239351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:51.239316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:51.364153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:51.364079 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:51.876211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:51.876177 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:54.637393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:54.637316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:54.765142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:54.765111 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:57.531198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:57.531171 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:57.650585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:57.650556 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:58.240825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:58.240792 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:58.357443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:58.357412 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:58.472396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:58.472351 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:58.631791 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:58.631721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:28:59.471008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:28:59.470977 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:00.249909 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:29:00.249878 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:29:01.838850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:01.838814 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:02.927839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:02.927799 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:03.075872 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:03.075831 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:03.202476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:03.202404 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:03.386549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:03.386516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:04.985995 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:04.985960 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:05.029622 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.029595 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w"] Nov 01 12:29:05.032667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.032650 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.120832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.120809 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a64131-4c62-4bdf-abb6-9c837f6dceba-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.120981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.120956 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqr8\" (UniqueName: \"kubernetes.io/projected/75a64131-4c62-4bdf-abb6-9c837f6dceba-kube-api-access-9wqr8\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.121059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.121029 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a64131-4c62-4bdf-abb6-9c837f6dceba-tmp\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.127031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.127013 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 12:29:05.127225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.127131 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-grtjg\"" Nov 01 12:29:05.127340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.127253 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 12:29:05.128377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.128355 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w"] Nov 01 12:29:05.222236 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.222211 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqr8\" (UniqueName: \"kubernetes.io/projected/75a64131-4c62-4bdf-abb6-9c837f6dceba-kube-api-access-9wqr8\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.222358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.222241 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a64131-4c62-4bdf-abb6-9c837f6dceba-tmp\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.222358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.222265 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a64131-4c62-4bdf-abb6-9c837f6dceba-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.222445 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:29:05.222430 4484 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Nov 01 12:29:05.222502 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:29:05.222491 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a64131-4c62-4bdf-abb6-9c837f6dceba-tls-certs podName:75a64131-4c62-4bdf-abb6-9c837f6dceba nodeName:}" failed. No retries permitted until 2026-11-01 12:29:05.722472518 +0000 UTC m=+415.472415479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/75a64131-4c62-4bdf-abb6-9c837f6dceba-tls-certs") pod "kube-auth-proxy-7bf7dc98fb-t2q7w" (UID: "75a64131-4c62-4bdf-abb6-9c837f6dceba") : secret "kube-auth-proxy-tls" not found Nov 01 12:29:05.224395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.224372 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a64131-4c62-4bdf-abb6-9c837f6dceba-tmp\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.278930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.278870 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqr8\" (UniqueName: \"kubernetes.io/projected/75a64131-4c62-4bdf-abb6-9c837f6dceba-kube-api-access-9wqr8\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.357941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.357904 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:05.491536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.491505 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:05.611660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.611586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:05.725880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.725846 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a64131-4c62-4bdf-abb6-9c837f6dceba-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.728285 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.728247 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a64131-4c62-4bdf-abb6-9c837f6dceba-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-t2q7w\" (UID: \"75a64131-4c62-4bdf-abb6-9c837f6dceba\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:05.905323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.905229 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:05.944470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:05.944439 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" Nov 01 12:29:06.304451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:06.304414 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w"] Nov 01 12:29:06.308661 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:29:06.308635 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a64131_4c62_4bdf_abb6_9c837f6dceba.slice/crio-e7b97bf186ed29dbc7a4c5fac9d68482f75a6a9d80b8c74436d5b7b197abc855 WatchSource:0}: Error finding container e7b97bf186ed29dbc7a4c5fac9d68482f75a6a9d80b8c74436d5b7b197abc855: Status 404 returned error can't find the container with id e7b97bf186ed29dbc7a4c5fac9d68482f75a6a9d80b8c74436d5b7b197abc855 Nov 01 12:29:06.761706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:06.761669 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" event={"ID":"75a64131-4c62-4bdf-abb6-9c837f6dceba","Type":"ContainerStarted","Data":"e7b97bf186ed29dbc7a4c5fac9d68482f75a6a9d80b8c74436d5b7b197abc855"} Nov 01 12:29:07.271160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:07.271126 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:07.636944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:07.636866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:07.877679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:07.877646 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:09.023789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:09.023747 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:09.160468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:09.160435 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:09.272482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:09.272448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:09.773792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:09.773759 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" event={"ID":"75a64131-4c62-4bdf-abb6-9c837f6dceba","Type":"ContainerStarted","Data":"dcf70b35fe1270761dc5f2c5ff0208fbd33d877d9871111c11b80a75039b5783"} Nov 01 12:29:09.880528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:09.880484 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-t2q7w" podStartSLOduration=3.027556804 podStartE2EDuration="5.880467603s" totalImagesPullingTime="2.852910799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:09.87975072 +0000 UTC m=+419.629693698" watchObservedRunningTime="2026-11-01 12:29:09.880467603 +0000 UTC m=+419.630410582" Nov 01 12:29:10.959183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:10.959114 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:11.073763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.073713 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:11.563401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.563364 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:29:11.567768 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.567750 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.620108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.620081 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ckzhr\"" Nov 01 12:29:11.620287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.620112 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 12:29:11.620287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.620160 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 12:29:11.628313 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.628278 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:11.666062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.666038 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 12:29:11.667148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.667124 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:29:11.671346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.671325 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a80816c-07b3-4b1a-9cf5-120d71f4a79b-config-file\") pod \"limitador-limitador-7c9b87d574-wmb4b\" (UID: \"3a80816c-07b3-4b1a-9cf5-120d71f4a79b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.671417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.671395 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxrx\" (UniqueName: \"kubernetes.io/projected/3a80816c-07b3-4b1a-9cf5-120d71f4a79b-kube-api-access-vtxrx\") pod \"limitador-limitador-7c9b87d574-wmb4b\" (UID: \"3a80816c-07b3-4b1a-9cf5-120d71f4a79b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.740541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.740521 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:11.772736 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.772712 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a80816c-07b3-4b1a-9cf5-120d71f4a79b-config-file\") pod \"limitador-limitador-7c9b87d574-wmb4b\" (UID: \"3a80816c-07b3-4b1a-9cf5-120d71f4a79b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.772821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.772765 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxrx\" (UniqueName: \"kubernetes.io/projected/3a80816c-07b3-4b1a-9cf5-120d71f4a79b-kube-api-access-vtxrx\") pod \"limitador-limitador-7c9b87d574-wmb4b\" (UID: \"3a80816c-07b3-4b1a-9cf5-120d71f4a79b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.773296 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.773281 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a80816c-07b3-4b1a-9cf5-120d71f4a79b-config-file\") pod \"limitador-limitador-7c9b87d574-wmb4b\" (UID: \"3a80816c-07b3-4b1a-9cf5-120d71f4a79b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.826431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.826382 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxrx\" (UniqueName: \"kubernetes.io/projected/3a80816c-07b3-4b1a-9cf5-120d71f4a79b-kube-api-access-vtxrx\") pod \"limitador-limitador-7c9b87d574-wmb4b\" (UID: \"3a80816c-07b3-4b1a-9cf5-120d71f4a79b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:11.849024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.849002 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:29:11.880167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:11.880146 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:12.237007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:12.236979 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:29:12.240262 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:29:12.240182 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a80816c_07b3_4b1a_9cf5_120d71f4a79b.slice/crio-1a6873ba430c26a78f239cacf2a021f33d50dc5467f647c3ac04d65479cadd4c WatchSource:0}: Error finding container 1a6873ba430c26a78f239cacf2a021f33d50dc5467f647c3ac04d65479cadd4c: Status 404 returned error can't find the container with id 1a6873ba430c26a78f239cacf2a021f33d50dc5467f647c3ac04d65479cadd4c Nov 01 12:29:12.784558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:12.784525 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" event={"ID":"3a80816c-07b3-4b1a-9cf5-120d71f4a79b","Type":"ContainerStarted","Data":"1a6873ba430c26a78f239cacf2a021f33d50dc5467f647c3ac04d65479cadd4c"} Nov 01 12:29:12.792454 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:12.792428 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:13.952520 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:13.952481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:14.067316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:14.067267 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:15.796603 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:15.796563 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" event={"ID":"3a80816c-07b3-4b1a-9cf5-120d71f4a79b","Type":"ContainerStarted","Data":"3f4cdb3a6450bba1d7e86f54027905922fa537ecb0a5fe80d2f049608103e12b"} Nov 01 12:29:15.797069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:15.796712 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:15.902863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:15.902820 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" podStartSLOduration=1.697771527 podStartE2EDuration="4.902807466s" totalImagesPullingTime="3.205035939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:15.90216142 +0000 UTC m=+425.652104400" watchObservedRunningTime="2026-11-01 12:29:15.902807466 +0000 UTC m=+425.652750480" Nov 01 12:29:19.152340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:19.152291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:19.266916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:19.266879 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:22.987189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:22.987151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:23.109690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:23.109647 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:26.800977 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:26.800947 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wmb4b" Nov 01 12:29:26.832371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:26.832343 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:26.970107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:26.970082 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:27.088391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:27.088294 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:30.250428 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:29:30.250393 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:29:30.260494 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:30.260468 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:30.388451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:30.388426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:31.017652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:31.017613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:41.673772 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:41.673721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:41.786291 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:41.786254 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:51.891431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:51.891395 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:52.004878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:52.004846 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:52.119671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:52.119636 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:52.229310 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:52.229281 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:52.345633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:52.345603 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:56.587653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:56.587618 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:56.772837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:56.772805 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:59.538089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:59.538056 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:29:59.825447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:29:59.825365 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:00.251128 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:30:00.251090 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:30:02.163739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:02.163693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:02.275710 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:02.275663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:18.963755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:18.963722 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q"] Nov 01 12:30:18.966857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:18.966841 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.068511 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.068436 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q"] Nov 01 12:30:19.070437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070413 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.070572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070453 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.070572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.070572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070497 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.070572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070540 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.070572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070560 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.070753 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.070602 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfpt9\" (UniqueName: \"kubernetes.io/projected/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-kube-api-access-hfpt9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.171871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171838 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.171871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171871 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.172112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171886 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfpt9\" (UniqueName: \"kubernetes.io/projected/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-kube-api-access-hfpt9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.172112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171924 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.172112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171959 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.172112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171975 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.172112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.171992 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.172733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.172704 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.174214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.174190 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.174488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.174468 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.174701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.174682 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.174774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.174730 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.225244 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.225194 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.225349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.225269 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfpt9\" (UniqueName: \"kubernetes.io/projected/f35ed7ef-9b70-4a47-85f6-c5496a308f2a-kube-api-access-hfpt9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-97k8q\" (UID: \"f35ed7ef-9b70-4a47-85f6-c5496a308f2a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.278243 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.278219 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:19.689097 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.689061 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q"] Nov 01 12:30:19.692606 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:30:19.692576 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35ed7ef_9b70_4a47_85f6_c5496a308f2a.slice/crio-802119f7e956b5b184ce5a41a59dc60d297ac73f00b3ee805ad4b1a65572c0d8 WatchSource:0}: Error finding container 802119f7e956b5b184ce5a41a59dc60d297ac73f00b3ee805ad4b1a65572c0d8: Status 404 returned error can't find the container with id 802119f7e956b5b184ce5a41a59dc60d297ac73f00b3ee805ad4b1a65572c0d8 Nov 01 12:30:19.694938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.694909 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:30:19.695019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.694968 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:30:19.874241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:19.874206 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:20.007691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:20.007658 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" event={"ID":"f35ed7ef-9b70-4a47-85f6-c5496a308f2a","Type":"ContainerStarted","Data":"c16453138ff1fbba05be740eafa876af1c8d093776669cb126cd8c786567509c"} Nov 01 12:30:20.008121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:20.007704 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" event={"ID":"f35ed7ef-9b70-4a47-85f6-c5496a308f2a","Type":"ContainerStarted","Data":"802119f7e956b5b184ce5a41a59dc60d297ac73f00b3ee805ad4b1a65572c0d8"} Nov 01 12:30:20.008121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:20.007909 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:20.009429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:20.009407 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-97k8q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:30:20.009542 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:20.009452 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" podUID="f35ed7ef-9b70-4a47-85f6-c5496a308f2a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:30:20.118653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:20.118600 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" podStartSLOduration=2.118582827 podStartE2EDuration="2.118582827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:20.117473163 +0000 UTC m=+489.867416152" watchObservedRunningTime="2026-11-01 12:30:20.118582827 +0000 UTC m=+489.868525808" Nov 01 12:30:21.012585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:21.012553 4484 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-97k8q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:30:21.012988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:21.012604 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" podUID="f35ed7ef-9b70-4a47-85f6-c5496a308f2a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:30:23.915651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:23.915564 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.013060 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.013037 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-97k8q" Nov 01 12:30:24.027402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.027372 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.138456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.138428 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.214948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.214874 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2"] Nov 01 12:30:24.215108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.215074 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerName="discovery" containerID="cri-o://6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd" gracePeriod=30 Nov 01 12:30:24.257312 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.257279 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.370007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.369970 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.456886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.456862 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:30:24.485828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.485807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.611428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.611390 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.616399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616363 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.616519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616435 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.616519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616465 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.616519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616495 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.616669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616544 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.616669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616583 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.616669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.616636 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts\") pod \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\" (UID: \"7359aaa2-ae08-460a-bc56-9fb7e1588fc9\") " Nov 01 12:30:24.617147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.617119 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:30:24.619247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.619218 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:30:24.619383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.619283 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "kube-api-access-67lr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:30:24.619711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.619675 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:30:24.619969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.619939 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:30:24.620055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.619981 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:30:24.620055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.620014 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig" pod "7359aaa2-ae08-460a-bc56-9fb7e1588fc9" (UID: "7359aaa2-ae08-460a-bc56-9fb7e1588fc9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:30:24.717501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717470 4484 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-dns-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.717501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717498 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.717501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717507 4484 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-cacerts\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.717715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717516 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lr7\" (UniqueName: \"kubernetes.io/projected/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-kube-api-access-67lr7\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.717715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717526 4484 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-local-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.717715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717535 4484 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-csr-ca-configmap\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.717715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.717544 4484 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7359aaa2-ae08-460a-bc56-9fb7e1588fc9-istio-kubeconfig\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:24.728191 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.728163 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.739339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.739262 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b5zjm"] Nov 01 12:30:24.739708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.739689 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerName="discovery" Nov 01 12:30:24.739752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.739712 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerName="discovery" Nov 01 12:30:24.739794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.739784 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerName="discovery" Nov 01 12:30:24.742886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.742868 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:24.791376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.791352 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 12:30:24.791502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.791446 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l9wf4\"" Nov 01 12:30:24.791551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.791538 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 12:30:24.791551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.791539 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 12:30:24.796382 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.796358 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b5zjm"] Nov 01 12:30:24.884146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.884105 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:24.919413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.919386 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2977l\" (UniqueName: \"kubernetes.io/projected/3f1e9d53-dcee-415e-83bb-bd112a38ee67-kube-api-access-2977l\") pod \"kserve-controller-manager-776cc66d47-b5zjm\" (UID: \"3f1e9d53-dcee-415e-83bb-bd112a38ee67\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:24.919748 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:24.919440 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f1e9d53-dcee-415e-83bb-bd112a38ee67-cert\") pod \"kserve-controller-manager-776cc66d47-b5zjm\" (UID: \"3f1e9d53-dcee-415e-83bb-bd112a38ee67\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:25.020076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.020000 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f1e9d53-dcee-415e-83bb-bd112a38ee67-cert\") pod \"kserve-controller-manager-776cc66d47-b5zjm\" (UID: \"3f1e9d53-dcee-415e-83bb-bd112a38ee67\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:25.020076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.020061 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2977l\" (UniqueName: \"kubernetes.io/projected/3f1e9d53-dcee-415e-83bb-bd112a38ee67-kube-api-access-2977l\") pod \"kserve-controller-manager-776cc66d47-b5zjm\" (UID: \"3f1e9d53-dcee-415e-83bb-bd112a38ee67\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:25.022437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.022420 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f1e9d53-dcee-415e-83bb-bd112a38ee67-cert\") pod \"kserve-controller-manager-776cc66d47-b5zjm\" (UID: \"3f1e9d53-dcee-415e-83bb-bd112a38ee67\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:25.027832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.027799 4484 generic.go:356] "Generic (PLEG): container finished" podID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" containerID="6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd" exitCode=0 Nov 01 12:30:25.027948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.027845 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" Nov 01 12:30:25.027948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.027880 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" event={"ID":"7359aaa2-ae08-460a-bc56-9fb7e1588fc9","Type":"ContainerDied","Data":"6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd"} Nov 01 12:30:25.027948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.027931 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2" event={"ID":"7359aaa2-ae08-460a-bc56-9fb7e1588fc9","Type":"ContainerDied","Data":"8e395a0ba6c8460f13b2fb1eb16a28e20b95f870f327e27364ed63855ca6a5a5"} Nov 01 12:30:25.028043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.027957 4484 scope.go:122] "RemoveContainer" containerID="6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd" Nov 01 12:30:25.036768 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.036749 4484 scope.go:122] "RemoveContainer" containerID="6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd" Nov 01 12:30:25.037027 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:30:25.037010 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd\": container with ID starting with 6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd not found: ID does not exist" containerID="6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd" Nov 01 12:30:25.037081 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.037036 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd"} err="failed to get container status \"6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd\": rpc error: code = NotFound desc = could not find container \"6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd\": container with ID starting with 6420ad704b8e6c894258841531cba727b3bb82b70db4de4e7b8234b67c8291cd not found: ID does not exist" Nov 01 12:30:25.072825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.072795 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2977l\" (UniqueName: \"kubernetes.io/projected/3f1e9d53-dcee-415e-83bb-bd112a38ee67-kube-api-access-2977l\") pod \"kserve-controller-manager-776cc66d47-b5zjm\" (UID: \"3f1e9d53-dcee-415e-83bb-bd112a38ee67\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:25.179323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.179282 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2"] Nov 01 12:30:25.184720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.184695 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lnkf2"] Nov 01 12:30:25.354395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.354327 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:25.712167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:25.712083 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b5zjm"] Nov 01 12:30:25.715633 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:30:25.715600 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1e9d53_dcee_415e_83bb_bd112a38ee67.slice/crio-ec862f5533bbbbc96b73d09e38fe7ea0145e58d874af70cf1ddfd83c214183f9 WatchSource:0}: Error finding container ec862f5533bbbbc96b73d09e38fe7ea0145e58d874af70cf1ddfd83c214183f9: Status 404 returned error can't find the container with id ec862f5533bbbbc96b73d09e38fe7ea0145e58d874af70cf1ddfd83c214183f9 Nov 01 12:30:26.015173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:26.015141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:26.032976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:26.032947 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" event={"ID":"3f1e9d53-dcee-415e-83bb-bd112a38ee67","Type":"ContainerStarted","Data":"ec862f5533bbbbc96b73d09e38fe7ea0145e58d874af70cf1ddfd83c214183f9"} Nov 01 12:30:26.367153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:26.367079 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7359aaa2-ae08-460a-bc56-9fb7e1588fc9" path="/var/lib/kubelet/pods/7359aaa2-ae08-460a-bc56-9fb7e1588fc9/volumes" Nov 01 12:30:26.921047 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:26.920996 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:27.052557 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:27.052525 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:29.046403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:29.046361 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" event={"ID":"3f1e9d53-dcee-415e-83bb-bd112a38ee67","Type":"ContainerStarted","Data":"88da5e3c261c3b142ae2d19f23a0741f568fb4d61e682c3364cca4b48c692f4e"} Nov 01 12:30:29.046904 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:29.046500 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:30:29.167986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:29.167935 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" podStartSLOduration=2.822726672 podStartE2EDuration="5.167921049s" totalImagesPullingTime="2.345194377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:29.165970621 +0000 UTC m=+498.915913598" watchObservedRunningTime="2026-11-01 12:30:29.167921049 +0000 UTC m=+498.917864027" Nov 01 12:30:29.257660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:29.257633 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:29.376747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:29.376666 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:29.576685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:29.576655 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:30.193289 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.193256 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6"] Nov 01 12:30:30.196461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.196445 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.249702 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:30:30.249673 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:30:30.296327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.296281 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 12:30:30.296327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.296291 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 12:30:30.296482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.296291 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 12:30:30.296482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.296291 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 12:30:30.296482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.296349 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-758k9\"" Nov 01 12:30:30.297615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.297601 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6"] Nov 01 12:30:30.358703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.358679 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvn4\" (UniqueName: \"kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.358829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.358718 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.358829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.358741 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.358829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.358770 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.459612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.459556 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.459612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.459581 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.459783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.459622 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvn4\" (UniqueName: \"kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.459783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.459668 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.460247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.460226 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.462082 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.462064 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.462170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.462116 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.511986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.511960 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvn4\" (UniqueName: \"kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4\") pod \"maas-api-6d669c7fcd-hf9b6\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.807941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.807911 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:30.847295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.847259 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:30.963863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:30.963806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.168375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.168290 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6"] Nov 01 12:30:31.168899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.168879 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.171860 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:30:31.171837 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f38950a_6d7c_49d5_8d22_d12d5cd26b4f.slice/crio-fc5e83cf8e87a17fe7ec04e752af0b0a2a4ddbcb2c418aa673428e8c0abd1c86 WatchSource:0}: Error finding container fc5e83cf8e87a17fe7ec04e752af0b0a2a4ddbcb2c418aa673428e8c0abd1c86: Status 404 returned error can't find the container with id fc5e83cf8e87a17fe7ec04e752af0b0a2a4ddbcb2c418aa673428e8c0abd1c86 Nov 01 12:30:31.293063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.293025 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.412175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.412141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.531561 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.531489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.660843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.660813 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.779728 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.779695 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:31.889286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:31.889216 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:32.000771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:32.000743 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:32.059387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:32.059355 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" event={"ID":"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f","Type":"ContainerStarted","Data":"fc5e83cf8e87a17fe7ec04e752af0b0a2a4ddbcb2c418aa673428e8c0abd1c86"} Nov 01 12:30:33.031484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:33.031448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:35.080176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:35.080131 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" event={"ID":"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f","Type":"ContainerStarted","Data":"c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd"} Nov 01 12:30:35.080679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:35.080275 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:35.186478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:35.186429 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" podStartSLOduration=2.052500566 podStartE2EDuration="5.186416088s" totalImagesPullingTime="3.133915522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:35.185053469 +0000 UTC m=+504.934996447" watchObservedRunningTime="2026-11-01 12:30:35.186416088 +0000 UTC m=+504.936359103" Nov 01 12:30:35.465970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:35.465890 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:30:41.089435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:41.089409 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:41.560910 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:41.560876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:41.854717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:41.854637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:42.528923 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.528888 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp"] Nov 01 12:30:42.532807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.532776 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546010 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546041 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546058 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546190 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546132 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlcl\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546190 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546174 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546242 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546289 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546267 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546291 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.546441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.546422 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.581033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.581014 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bc8bt\"" Nov 01 12:30:42.628797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.628770 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp"] Nov 01 12:30:42.647155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647124 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647166 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647202 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647225 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647254 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlcl\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647423 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647508 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647554 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647585 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647853 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647828 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.647921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647855 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.648011 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.647983 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.648146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.648125 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.648391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.648374 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.649835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.649812 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.649936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.649920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.702157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.702130 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.702327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.702130 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlcl\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl\") pod \"maas-default-gateway-openshift-default-7df854bf67-sdqqp\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:42.844662 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:42.844586 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:43.254818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:43.254785 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp"] Nov 01 12:30:43.257699 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:30:43.257666 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a531eb_ae89_4ab2_9aac_029c6da7c74e.slice/crio-45f5e6829f15c90fddd8609f1cf0e9f06fa8db1a66df8e95239f62d77d528ca4 WatchSource:0}: Error finding container 45f5e6829f15c90fddd8609f1cf0e9f06fa8db1a66df8e95239f62d77d528ca4: Status 404 returned error can't find the container with id 45f5e6829f15c90fddd8609f1cf0e9f06fa8db1a66df8e95239f62d77d528ca4 Nov 01 12:30:44.116019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:44.115979 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" event={"ID":"39a531eb-ae89-4ab2-9aac-029c6da7c74e","Type":"ContainerStarted","Data":"45f5e6829f15c90fddd8609f1cf0e9f06fa8db1a66df8e95239f62d77d528ca4"} Nov 01 12:30:47.205166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:47.205129 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:30:47.205563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:47.205247 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:30:47.205563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:47.205316 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:30:47.477549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:47.477509 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:30:48.134007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:48.133969 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" event={"ID":"39a531eb-ae89-4ab2-9aac-029c6da7c74e","Type":"ContainerStarted","Data":"5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5"} Nov 01 12:30:48.334514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:48.334458 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podStartSLOduration=2.389809606 podStartE2EDuration="6.33444228s" totalImagesPullingTime="3.944632674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:48.287362975 +0000 UTC m=+518.037305953" watchObservedRunningTime="2026-11-01 12:30:48.33444228 +0000 UTC m=+518.084385258" Nov 01 12:30:48.852018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:48.851986 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:48.852348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:48.852325 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sdqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 01 12:30:48.852529 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:48.852501 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 01 12:30:49.857326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:49.851637 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sdqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 01 12:30:49.857326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:49.851699 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 01 12:30:50.849324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:50.845665 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sdqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 01 12:30:50.849324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:50.845728 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 01 12:30:51.856325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:51.849660 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sdqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 01 12:30:51.856325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:51.849723 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 01 12:30:52.487661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.487631 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6"] Nov 01 12:30:52.487840 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.487815 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" podUID="8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" containerName="maas-api" containerID="cri-o://c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd" gracePeriod=30 Nov 01 12:30:52.776082 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.776015 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:52.830623 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.830592 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls\") pod \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " Nov 01 12:30:52.830797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.830633 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca\") pod \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " Nov 01 12:30:52.830797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.830675 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4\" (UniqueName: \"kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4\") pod \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " Nov 01 12:30:52.830797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.830716 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs\") pod \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\" (UID: \"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f\") " Nov 01 12:30:52.831040 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.831010 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca" pod "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" (UID: "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:30:52.832933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.832907 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls" pod "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" (UID: "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:30:52.833022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.832929 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs" pod "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" (UID: "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:30:52.833022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.832959 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4" pod "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" (UID: "8f38950a-6d7c-49d5-8d22-d12d5cd26b4f"). InnerVolumeSpecName "kube-api-access-9xvn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:30:52.865229 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.864756 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sdqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Nov 01 12:30:52.865229 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.864821 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Nov 01 12:30:52.931989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.931955 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:52.931989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.931981 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:52.931989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.931992 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:52.932207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:52.932001 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xvn4\" (UniqueName: \"kubernetes.io/projected/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f-kube-api-access-9xvn4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:30:53.181796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.181700 4484 generic.go:356] "Generic (PLEG): container finished" podID="8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" containerID="c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd" exitCode=0 Nov 01 12:30:53.181796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.181771 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" event={"ID":"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f","Type":"ContainerDied","Data":"c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd"} Nov 01 12:30:53.182018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.181775 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" Nov 01 12:30:53.182018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.181869 4484 scope.go:122] "RemoveContainer" containerID="c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd" Nov 01 12:30:53.182018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.181856 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6" event={"ID":"8f38950a-6d7c-49d5-8d22-d12d5cd26b4f","Type":"ContainerDied","Data":"fc5e83cf8e87a17fe7ec04e752af0b0a2a4ddbcb2c418aa673428e8c0abd1c86"} Nov 01 12:30:53.191454 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.191431 4484 scope.go:122] "RemoveContainer" containerID="c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd" Nov 01 12:30:53.191808 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:30:53.191785 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd\": container with ID starting with c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd not found: ID does not exist" containerID="c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd" Nov 01 12:30:53.191896 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.191819 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd"} err="failed to get container status \"c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd\": rpc error: code = NotFound desc = could not find container \"c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd\": container with ID starting with c14e5bc10667c4fc7483ce58e5a6f918ff23b9af5d255053de7eedca68f674fd not found: ID does not exist" Nov 01 12:30:53.351351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.351316 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6"] Nov 01 12:30:53.354210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.354186 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hf9b6"] Nov 01 12:30:53.849391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.849364 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:53.849658 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.849626 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:53.850219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:53.850204 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:30:54.366534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:54.366502 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" path="/var/lib/kubelet/pods/8f38950a-6d7c-49d5-8d22-d12d5cd26b4f/volumes" Nov 01 12:30:59.649444 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:30:59.649410 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:31:00.056847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:00.056820 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b5zjm" Nov 01 12:31:00.250586 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:31:00.250540 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:31:02.505982 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:02.505951 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:31:12.764344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.764313 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj"] Nov 01 12:31:12.764763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.764672 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" containerName="maas-api" Nov 01 12:31:12.764763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.764687 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" containerName="maas-api" Nov 01 12:31:12.764763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.764745 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f38950a-6d7c-49d5-8d22-d12d5cd26b4f" containerName="maas-api" Nov 01 12:31:12.770931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.770911 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897262 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897471 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897326 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv86q\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897471 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897399 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897477 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897528 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897546 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897568 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897609 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.897711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.897680 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.956884 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.956844 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj"] Nov 01 12:31:12.998422 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998392 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998429 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998451 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv86q\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998467 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998601 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998673 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998704 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998739 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.998833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998773 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.999028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.998900 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.999113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.999082 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.999239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.999159 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.999239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.999175 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:12.999389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:12.999371 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:13.000931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.000912 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:13.001333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.001293 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:13.052100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.052032 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv86q\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:13.052592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.052573 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-szcjj\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:13.083453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.083420 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:13.494622 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.494594 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj"] Nov 01 12:31:13.497680 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:31:13.497652 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49ef56a_ea03_49cb_87a6_9c277eb01420.slice/crio-0f6846ba3495f6d6bb087d0e803dd5ab763a02c24fd5d592c7b7df65d6a5360e WatchSource:0}: Error finding container 0f6846ba3495f6d6bb087d0e803dd5ab763a02c24fd5d592c7b7df65d6a5360e: Status 404 returned error can't find the container with id 0f6846ba3495f6d6bb087d0e803dd5ab763a02c24fd5d592c7b7df65d6a5360e Nov 01 12:31:13.499976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.499944 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:31:13.500035 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.500012 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:31:13.500080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:13.500048 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:31:14.270927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:14.270881 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" event={"ID":"e49ef56a-ea03-49cb-87a6-9c277eb01420","Type":"ContainerStarted","Data":"6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344"} Nov 01 12:31:14.271348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:14.270938 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" event={"ID":"e49ef56a-ea03-49cb-87a6-9c277eb01420","Type":"ContainerStarted","Data":"0f6846ba3495f6d6bb087d0e803dd5ab763a02c24fd5d592c7b7df65d6a5360e"} Nov 01 12:31:14.380670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:14.380508 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podStartSLOduration=2.380489769 podStartE2EDuration="2.380489769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:14.379922438 +0000 UTC m=+544.129865422" watchObservedRunningTime="2026-11-01 12:31:14.380489769 +0000 UTC m=+544.130432748" Nov 01 12:31:15.083836 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:15.083796 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:15.085178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:15.085154 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-szcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 01 12:31:15.085281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:15.085203 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 01 12:31:16.084414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:16.084366 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-szcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 01 12:31:16.084807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:16.084431 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 01 12:31:17.083996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:17.083956 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-szcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 01 12:31:17.084215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:17.084019 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 01 12:31:18.084847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:18.084807 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-szcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 01 12:31:18.085368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:18.084870 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 01 12:31:19.084164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:19.084123 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-szcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 01 12:31:19.084404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:19.084200 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 01 12:31:20.087947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:20.087913 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:20.088431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:20.088295 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:20.089027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:20.089005 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:31:27.861650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:27.861617 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf"] Nov 01 12:31:27.864628 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:27.864612 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:27.959856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:27.959822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf"] Nov 01 12:31:28.048457 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048419 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048474 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048533 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048613 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048642 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048659 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048680 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048812 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.048848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.048832 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97t57\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.149954 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97t57\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.149990 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150021 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150041 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150086 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150119 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150147 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150176 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150227 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150474 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150541 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150663 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.150749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.150706 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.151027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.151005 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.152456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.152430 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.152659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.152643 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.203177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.203142 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.203339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.203316 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97t57\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57\") pod \"maas-default-gateway-openshift-default-7df854bf67-bzhdf\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.477218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.477188 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:28.886193 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.886112 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf"] Nov 01 12:31:28.889286 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:31:28.889258 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b76a5a1_fa75_4b8e_ae12_64195448c023.slice/crio-126205c3faabb4c83531c7a86ad47025d401cbc49bdfbffae6b443cb1736feab WatchSource:0}: Error finding container 126205c3faabb4c83531c7a86ad47025d401cbc49bdfbffae6b443cb1736feab: Status 404 returned error can't find the container with id 126205c3faabb4c83531c7a86ad47025d401cbc49bdfbffae6b443cb1736feab Nov 01 12:31:28.891542 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.891508 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:31:28.891607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.891572 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:31:28.891645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:28.891609 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:31:29.331376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:29.331328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" event={"ID":"2b76a5a1-fa75-4b8e-ae12-64195448c023","Type":"ContainerStarted","Data":"30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db"} Nov 01 12:31:29.331582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:29.331386 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" event={"ID":"2b76a5a1-fa75-4b8e-ae12-64195448c023","Type":"ContainerStarted","Data":"126205c3faabb4c83531c7a86ad47025d401cbc49bdfbffae6b443cb1736feab"} Nov 01 12:31:29.440437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:29.440380 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podStartSLOduration=2.44036631 podStartE2EDuration="2.44036631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:29.438896754 +0000 UTC m=+559.188839729" watchObservedRunningTime="2026-11-01 12:31:29.44036631 +0000 UTC m=+559.190309287" Nov 01 12:31:29.477513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:29.477482 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:29.478828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:29.478802 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 12:31:29.478938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:29.478847 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 12:31:30.275390 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:31:30.275355 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:31:30.477545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:30.477506 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 12:31:30.477709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:30.477560 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 12:31:31.478316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:31.478273 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 12:31:31.478872 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:31.478345 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 12:31:32.477637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:32.477604 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 12:31:32.477813 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:32.477658 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 12:31:33.477793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:33.477759 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 12:31:33.478172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:33.477817 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 12:31:34.477841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:34.477800 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 12:31:34.478249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:34.477856 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 12:31:35.481649 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:35.481618 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:35.482087 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:35.481913 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:35.482640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:35.482622 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:31:45.893014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:31:45.892977 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:00.250470 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:32:00.250364 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:32:02.824060 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:02.824026 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:21.076074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:21.076038 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:21.209644 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:21.209611 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:30.250873 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:32:30.250834 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:32:41.881867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.881833 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mrlsf"] Nov 01 12:32:41.885080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.885064 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.925695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.925662 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4fa5ee6-c7c7-419d-9370-5048be88c10d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.925695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.925699 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7s7z\" (UniqueName: \"kubernetes.io/projected/b4fa5ee6-c7c7-419d-9370-5048be88c10d-kube-api-access-s7s7z\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.925921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.925857 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.925921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.925902 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.926028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.925976 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.926086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.926030 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-home\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:41.962271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.962223 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:41.979463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.979423 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5s8np\"" Nov 01 12:32:41.979463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.979423 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 12:32:41.979463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.979423 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 12:32:41.979689 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.979463 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 12:32:41.981769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:41.981743 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mrlsf"] Nov 01 12:32:42.027162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.027110 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.027162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.027160 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.028133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.027206 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.028133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.027326 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-home\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.028133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.027399 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4fa5ee6-c7c7-419d-9370-5048be88c10d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.028133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.027430 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7s7z\" (UniqueName: \"kubernetes.io/projected/b4fa5ee6-c7c7-419d-9370-5048be88c10d-kube-api-access-s7s7z\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.028961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.028575 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-home\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.029085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.029070 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.029214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.029187 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.030939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.030913 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4fa5ee6-c7c7-419d-9370-5048be88c10d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.031650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.031607 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4fa5ee6-c7c7-419d-9370-5048be88c10d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.080607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.080573 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7s7z\" (UniqueName: \"kubernetes.io/projected/b4fa5ee6-c7c7-419d-9370-5048be88c10d-kube-api-access-s7s7z\") pod \"echo-simulator-kserve-64b6f8f96c-mrlsf\" (UID: \"b4fa5ee6-c7c7-419d-9370-5048be88c10d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.197065 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.196965 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:42.567577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.567542 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mrlsf"] Nov 01 12:32:42.570649 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:32:42.570617 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fa5ee6_c7c7_419d_9370_5048be88c10d.slice/crio-d76bf26512708cbc0a2735dba0a9259a96ab15528a15208d14f1d40f4dbd129f WatchSource:0}: Error finding container d76bf26512708cbc0a2735dba0a9259a96ab15528a15208d14f1d40f4dbd129f: Status 404 returned error can't find the container with id d76bf26512708cbc0a2735dba0a9259a96ab15528a15208d14f1d40f4dbd129f Nov 01 12:32:42.572037 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.572016 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:32:42.622237 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:42.622207 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" event={"ID":"b4fa5ee6-c7c7-419d-9370-5048be88c10d","Type":"ContainerStarted","Data":"d76bf26512708cbc0a2735dba0a9259a96ab15528a15208d14f1d40f4dbd129f"} Nov 01 12:32:45.123369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.123278 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh"] Nov 01 12:32:45.126642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.126625 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.160228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.160191 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.160400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.160247 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.160400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.160321 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.160485 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.160410 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.160485 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.160462 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.160485 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.160481 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdj5\" (UniqueName: \"kubernetes.io/projected/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-kube-api-access-jhdj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.175399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.175375 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 12:32:45.200578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.200556 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:45.222419 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.222399 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh"] Nov 01 12:32:45.227515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.227494 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:32:45.261367 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261343 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.261459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261373 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.261459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261390 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.261540 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261471 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.261540 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261532 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.261624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261551 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdj5\" (UniqueName: \"kubernetes.io/projected/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-kube-api-access-jhdj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.261775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261747 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.262005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261973 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.262005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.261989 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.263711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.263686 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.263937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.263921 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.316807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.316769 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdj5\" (UniqueName: \"kubernetes.io/projected/b95d7585-b83b-4e4a-bfac-cf3c16c721d3-kube-api-access-jhdj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh\" (UID: \"b95d7585-b83b-4e4a-bfac-cf3c16c721d3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.447972 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.447870 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:45.639039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.638999 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" event={"ID":"b4fa5ee6-c7c7-419d-9370-5048be88c10d","Type":"ContainerStarted","Data":"d19e09f7cbf63aeb63b5f43aed5e996ac5db571e9cabeeb627d938bd55a73df7"} Nov 01 12:32:45.856923 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.856797 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" podStartSLOduration=2.574638072 podStartE2EDuration="4.856776856s" totalImagesPullingTime="2.282138784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:32:45.757407476 +0000 UTC m=+635.507350455" watchObservedRunningTime="2026-11-01 12:32:45.856776856 +0000 UTC m=+635.606719835" Nov 01 12:32:45.857717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:45.857690 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh"] Nov 01 12:32:45.860993 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:32:45.860965 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb95d7585_b83b_4e4a_bfac_cf3c16c721d3.slice/crio-e044e913107f7146177c006b7e9d6e94b9f415f6d8f43b8cb1badad9ed78f31a WatchSource:0}: Error finding container e044e913107f7146177c006b7e9d6e94b9f415f6d8f43b8cb1badad9ed78f31a: Status 404 returned error can't find the container with id e044e913107f7146177c006b7e9d6e94b9f415f6d8f43b8cb1badad9ed78f31a Nov 01 12:32:46.198136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:46.198058 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:46.644319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:46.644270 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" event={"ID":"b95d7585-b83b-4e4a-bfac-cf3c16c721d3","Type":"ContainerStarted","Data":"e044e913107f7146177c006b7e9d6e94b9f415f6d8f43b8cb1badad9ed78f31a"} Nov 01 12:32:48.653316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:48.653268 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" event={"ID":"b95d7585-b83b-4e4a-bfac-cf3c16c721d3","Type":"ContainerStarted","Data":"847117696c15b7aee3926bfcc47a9b68904105309454e9017bce21e31544435f"} Nov 01 12:32:48.760727 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:48.760671 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podStartSLOduration=1.74229835 podStartE2EDuration="3.760657186s" totalImagesPullingTime="2.018358836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:32:48.75918566 +0000 UTC m=+638.509128640" watchObservedRunningTime="2026-11-01 12:32:48.760657186 +0000 UTC m=+638.510600142" Nov 01 12:32:49.448123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:49.448085 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:49.787651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:49.787613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:51.795188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:51.795160 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:51.908578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:51.908545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:52.198082 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:52.198001 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:52.210811 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:52.210782 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:52.681024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:52.680997 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mrlsf" Nov 01 12:32:53.109809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.109776 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4"] Nov 01 12:32:53.113239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.113222 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.149968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.149934 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.150264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.150236 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vk9\" (UniqueName: \"kubernetes.io/projected/5454642c-8004-4031-ae58-5f6910da0599-kube-api-access-89vk9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.150429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.150411 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.150785 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.150760 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5454642c-8004-4031-ae58-5f6910da0599-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.150890 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.150872 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.151009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.150904 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.161401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.161373 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 12:32:53.182832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.182794 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:32:53.208851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.208820 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4"] Nov 01 12:32:53.251534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.251496 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.251672 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.251553 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vk9\" (UniqueName: \"kubernetes.io/projected/5454642c-8004-4031-ae58-5f6910da0599-kube-api-access-89vk9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.251672 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.251607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.251672 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.251626 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5454642c-8004-4031-ae58-5f6910da0599-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.251842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.251678 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.251842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.251699 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.252096 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.252073 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.252197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.252101 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.252197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.252144 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.254656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.254632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5454642c-8004-4031-ae58-5f6910da0599-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.259556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.259529 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5454642c-8004-4031-ae58-5f6910da0599-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.305854 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.305819 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vk9\" (UniqueName: \"kubernetes.io/projected/5454642c-8004-4031-ae58-5f6910da0599-kube-api-access-89vk9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wpvz4\" (UID: \"5454642c-8004-4031-ae58-5f6910da0599\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.424738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.424665 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:53.802715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:53.802681 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4"] Nov 01 12:32:53.806107 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:32:53.806083 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5454642c_8004_4031_ae58_5f6910da0599.slice/crio-8d6d1c68e0402a758b5ec1f238c65847dfecdaedff98c757d43cdbafcf923f3d WatchSource:0}: Error finding container 8d6d1c68e0402a758b5ec1f238c65847dfecdaedff98c757d43cdbafcf923f3d: Status 404 returned error can't find the container with id 8d6d1c68e0402a758b5ec1f238c65847dfecdaedff98c757d43cdbafcf923f3d Nov 01 12:32:54.678952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:54.678862 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" event={"ID":"5454642c-8004-4031-ae58-5f6910da0599","Type":"ContainerStarted","Data":"4f0505623c190b236545f84578afe7a3a6f9fd741d7263225df86241370ce460"} Nov 01 12:32:54.678952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:54.678946 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" event={"ID":"5454642c-8004-4031-ae58-5f6910da0599","Type":"ContainerStarted","Data":"8d6d1c68e0402a758b5ec1f238c65847dfecdaedff98c757d43cdbafcf923f3d"} Nov 01 12:32:54.785084 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:54.785040 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" podStartSLOduration=1.2136384279999999 podStartE2EDuration="1.785026938s" totalImagesPullingTime="571.38851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:32:54.78475818 +0000 UTC m=+644.534701155" watchObservedRunningTime="2026-11-01 12:32:54.785026938 +0000 UTC m=+644.534969917" Nov 01 12:32:55.425556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:55.425519 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:32:55.448626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:55.448601 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:55.460855 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:55.460831 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:55.560105 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:55.560078 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:32:55.696264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:55.696178 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:32:57.939526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:32:57.939497 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:00.251208 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:33:00.251173 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:33:02.256161 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.256126 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p"] Nov 01 12:33:02.259519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.259496 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.337318 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.337272 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:02.340393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.340372 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4z72\" (UniqueName: \"kubernetes.io/projected/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-kube-api-access-b4z72\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.340476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.340431 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.340476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.340455 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.340555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.340518 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.340555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.340548 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.340631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.340581 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.357225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.357201 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 12:33:02.358718 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.358698 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p"] Nov 01 12:33:02.441458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441424 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.441630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441474 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.441630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441502 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4z72\" (UniqueName: \"kubernetes.io/projected/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-kube-api-access-b4z72\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.441630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441549 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.441755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441645 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.441755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441699 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.441827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441813 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.442003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441976 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.442113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.441986 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.443691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.443664 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.443990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.443970 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.494867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.494836 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4z72\" (UniqueName: \"kubernetes.io/projected/28c4ca14-c2bc-4d22-82b9-0adc9acadf02-kube-api-access-b4z72\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7jk9p\" (UID: \"28c4ca14-c2bc-4d22-82b9-0adc9acadf02\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.570967 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.570876 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:02.939598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:02.939465 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p"] Nov 01 12:33:02.942650 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:33:02.942613 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c4ca14_c2bc_4d22_82b9_0adc9acadf02.slice/crio-e6c1c693b55084f773c15ec11de307b58d25ddf637fb174b91e530b3ed25d40a WatchSource:0}: Error finding container e6c1c693b55084f773c15ec11de307b58d25ddf637fb174b91e530b3ed25d40a: Status 404 returned error can't find the container with id e6c1c693b55084f773c15ec11de307b58d25ddf637fb174b91e530b3ed25d40a Nov 01 12:33:03.165358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:03.165293 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:03.364323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:03.363978 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:03.425717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:03.425667 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:33:03.438456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:03.438430 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:33:03.726685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:03.726616 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" event={"ID":"28c4ca14-c2bc-4d22-82b9-0adc9acadf02","Type":"ContainerStarted","Data":"e6c1c693b55084f773c15ec11de307b58d25ddf637fb174b91e530b3ed25d40a"} Nov 01 12:33:03.740564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:03.740544 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wpvz4" Nov 01 12:33:04.732954 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:04.732913 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" event={"ID":"28c4ca14-c2bc-4d22-82b9-0adc9acadf02","Type":"ContainerStarted","Data":"38b103af7df5708e4d57617506add099fef0bcb0072061f86336e8142dcedf4f"} Nov 01 12:33:04.840216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:04.840160 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" podStartSLOduration=2.215253069 podStartE2EDuration="2.840146623s" totalImagesPullingTime="624.893554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:04.838082065 +0000 UTC m=+654.588025044" watchObservedRunningTime="2026-11-01 12:33:04.840146623 +0000 UTC m=+654.590089602" Nov 01 12:33:05.571900 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:05.571865 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:06.525952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:06.525918 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:08.461507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:08.461465 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:33:12.571766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:12.571723 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:12.584652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:12.584628 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:12.837934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:12.837845 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7jk9p" Nov 01 12:33:13.277505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:13.277471 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:22.803420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:22.803384 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:33:30.250063 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:33:30.249982 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:33:41.625990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.625954 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:41.705002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.702835 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-zblbl"] Nov 01 12:33:41.710055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.709960 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:41.759226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.759198 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-zblbl"] Nov 01 12:33:41.806339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.806309 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-jxdck\"" Nov 01 12:33:41.806459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.806347 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 12:33:41.806459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.806347 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 12:33:41.810872 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.810820 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs\") pod \"ai-gateway-controller-697b76b867-zblbl\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:41.810872 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.810858 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtv9l\" (UniqueName: \"kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l\") pod \"ai-gateway-controller-697b76b867-zblbl\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:41.866799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.866765 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:41.912143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.912088 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs\") pod \"ai-gateway-controller-697b76b867-zblbl\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:41.912143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.912116 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtv9l\" (UniqueName: \"kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l\") pod \"ai-gateway-controller-697b76b867-zblbl\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:41.914450 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.914432 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs\") pod \"ai-gateway-controller-697b76b867-zblbl\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:41.964205 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:41.964182 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtv9l\" (UniqueName: \"kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l\") pod \"ai-gateway-controller-697b76b867-zblbl\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:42.022640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:42.022614 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:42.382553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:42.382471 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-zblbl"] Nov 01 12:33:42.385285 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:33:42.385259 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aff065_d521_413b_b054_4730e488c301.slice/crio-f8397655b976c99aaae2becd8208e975e2f73f2bcf26b272f5542b9f524d7aa3 WatchSource:0}: Error finding container f8397655b976c99aaae2becd8208e975e2f73f2bcf26b272f5542b9f524d7aa3: Status 404 returned error can't find the container with id f8397655b976c99aaae2becd8208e975e2f73f2bcf26b272f5542b9f524d7aa3 Nov 01 12:33:42.964092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:42.964052 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" event={"ID":"34aff065-d521-413b-b054-4730e488c301","Type":"ContainerStarted","Data":"f8397655b976c99aaae2becd8208e975e2f73f2bcf26b272f5542b9f524d7aa3"} Nov 01 12:33:48.988915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:48.988882 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" event={"ID":"34aff065-d521-413b-b054-4730e488c301","Type":"ContainerStarted","Data":"31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984"} Nov 01 12:33:48.989394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:48.988972 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:33:49.094648 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:49.094546 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" podStartSLOduration=1.7006839070000002 podStartE2EDuration="8.094533786s" totalImagesPullingTime="6.393849879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:49.093976566 +0000 UTC m=+698.843919540" watchObservedRunningTime="2026-11-01 12:33:49.094533786 +0000 UTC m=+698.844476793" Nov 01 12:33:49.724939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:49.724906 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:52.636696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:52.636667 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:52.856045 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:52.856012 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:33:53.061650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.061623 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qbnms"] Nov 01 12:33:53.064899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.064884 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.113230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.113204 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 12:33:53.159427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.159392 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ccfkx\"" Nov 01 12:33:53.161000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.160969 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qbnms"] Nov 01 12:33:53.213099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.213060 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c72fad4-669b-426c-8c56-153c26c4608b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qbnms\" (UID: \"4c72fad4-669b-426c-8c56-153c26c4608b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.213286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.213116 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87ls\" (UniqueName: \"kubernetes.io/projected/4c72fad4-669b-426c-8c56-153c26c4608b-kube-api-access-f87ls\") pod \"payload-pre-processing-7b98bb678c-qbnms\" (UID: \"4c72fad4-669b-426c-8c56-153c26c4608b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.214426 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.214400 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ph5fh"] Nov 01 12:33:53.218994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.218971 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.270012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.269981 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ph5fh"] Nov 01 12:33:53.314464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.314397 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbjz\" (UniqueName: \"kubernetes.io/projected/2478460e-643b-4c88-ba89-bfa8fbc9a2ae-kube-api-access-5zbjz\") pod \"payload-processing-55fbbbf9dc-ph5fh\" (UID: \"2478460e-643b-4c88-ba89-bfa8fbc9a2ae\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.314464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.314455 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c72fad4-669b-426c-8c56-153c26c4608b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qbnms\" (UID: \"4c72fad4-669b-426c-8c56-153c26c4608b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.314630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.314510 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2478460e-643b-4c88-ba89-bfa8fbc9a2ae-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ph5fh\" (UID: \"2478460e-643b-4c88-ba89-bfa8fbc9a2ae\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.314630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.314563 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87ls\" (UniqueName: \"kubernetes.io/projected/4c72fad4-669b-426c-8c56-153c26c4608b-kube-api-access-f87ls\") pod \"payload-pre-processing-7b98bb678c-qbnms\" (UID: \"4c72fad4-669b-426c-8c56-153c26c4608b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.315251 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.315234 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c72fad4-669b-426c-8c56-153c26c4608b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qbnms\" (UID: \"4c72fad4-669b-426c-8c56-153c26c4608b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.367292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.367263 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87ls\" (UniqueName: \"kubernetes.io/projected/4c72fad4-669b-426c-8c56-153c26c4608b-kube-api-access-f87ls\") pod \"payload-pre-processing-7b98bb678c-qbnms\" (UID: \"4c72fad4-669b-426c-8c56-153c26c4608b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.376970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.376941 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:53.415930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.415902 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbjz\" (UniqueName: \"kubernetes.io/projected/2478460e-643b-4c88-ba89-bfa8fbc9a2ae-kube-api-access-5zbjz\") pod \"payload-processing-55fbbbf9dc-ph5fh\" (UID: \"2478460e-643b-4c88-ba89-bfa8fbc9a2ae\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.416035 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.415950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2478460e-643b-4c88-ba89-bfa8fbc9a2ae-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ph5fh\" (UID: \"2478460e-643b-4c88-ba89-bfa8fbc9a2ae\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.416590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.416571 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2478460e-643b-4c88-ba89-bfa8fbc9a2ae-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ph5fh\" (UID: \"2478460e-643b-4c88-ba89-bfa8fbc9a2ae\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.468501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.468469 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbjz\" (UniqueName: \"kubernetes.io/projected/2478460e-643b-4c88-ba89-bfa8fbc9a2ae-kube-api-access-5zbjz\") pod \"payload-processing-55fbbbf9dc-ph5fh\" (UID: \"2478460e-643b-4c88-ba89-bfa8fbc9a2ae\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.546325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.546285 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:53.735451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.735264 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qbnms"] Nov 01 12:33:53.738877 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:33:53.738846 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c72fad4_669b_426c_8c56_153c26c4608b.slice/crio-d03e7e5fb93507765726a2f460d9e655dec694482c460b1f9cfe595e38713b08 WatchSource:0}: Error finding container d03e7e5fb93507765726a2f460d9e655dec694482c460b1f9cfe595e38713b08: Status 404 returned error can't find the container with id d03e7e5fb93507765726a2f460d9e655dec694482c460b1f9cfe595e38713b08 Nov 01 12:33:53.905386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:53.905329 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ph5fh"] Nov 01 12:33:53.907966 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:33:53.907937 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2478460e_643b_4c88_ba89_bfa8fbc9a2ae.slice/crio-927a38262b8765ec980efecfc4e8debcf039cab8c4df9937b78fcb716549a326 WatchSource:0}: Error finding container 927a38262b8765ec980efecfc4e8debcf039cab8c4df9937b78fcb716549a326: Status 404 returned error can't find the container with id 927a38262b8765ec980efecfc4e8debcf039cab8c4df9937b78fcb716549a326 Nov 01 12:33:54.011338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:54.011295 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" event={"ID":"4c72fad4-669b-426c-8c56-153c26c4608b","Type":"ContainerStarted","Data":"d03e7e5fb93507765726a2f460d9e655dec694482c460b1f9cfe595e38713b08"} Nov 01 12:33:54.012589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:54.012565 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" event={"ID":"2478460e-643b-4c88-ba89-bfa8fbc9a2ae","Type":"ContainerStarted","Data":"927a38262b8765ec980efecfc4e8debcf039cab8c4df9937b78fcb716549a326"} Nov 01 12:33:57.029752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:57.029720 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" event={"ID":"2478460e-643b-4c88-ba89-bfa8fbc9a2ae","Type":"ContainerStarted","Data":"f8c41131221da17cbda73e52a919115d6d3c70e1bfefcbe6e7b4c3c9cddf377b"} Nov 01 12:33:57.030170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:57.029816 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:33:57.135826 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:57.134994 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" podStartSLOduration=1.331860673 podStartE2EDuration="4.13497766s" totalImagesPullingTime="2.803116987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:57.133908531 +0000 UTC m=+706.883851521" watchObservedRunningTime="2026-11-01 12:33:57.13497766 +0000 UTC m=+706.884920640" Nov 01 12:33:58.035278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:58.035236 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" event={"ID":"4c72fad4-669b-426c-8c56-153c26c4608b","Type":"ContainerStarted","Data":"49d5ac2edb6d99b5ed2164fd558ec8fef47bf4b106d20418d919f2acedb165d6"} Nov 01 12:33:58.035748 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:58.035383 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:33:58.140217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:58.140159 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" podStartSLOduration=1.7750918919999998 podStartE2EDuration="5.140146091s" totalImagesPullingTime="3.365054199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:58.13938349 +0000 UTC m=+707.889326467" watchObservedRunningTime="2026-11-01 12:33:58.140146091 +0000 UTC m=+707.890089069" Nov 01 12:33:59.996578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:33:59.996550 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:34:00.249526 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:34:00.249448 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:34:00.330070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.330041 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-549c4cbc45-hsm9p"] Nov 01 12:34:00.333521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.333504 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.387122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.387090 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-549c4cbc45-hsm9p"] Nov 01 12:34:00.488426 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.488398 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g48b\" (UniqueName: \"kubernetes.io/projected/9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b-kube-api-access-8g48b\") pod \"ai-gateway-controller-549c4cbc45-hsm9p\" (UID: \"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b\") " pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.488551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.488447 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b-metrics-certs\") pod \"ai-gateway-controller-549c4cbc45-hsm9p\" (UID: \"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b\") " pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.589675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.589614 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g48b\" (UniqueName: \"kubernetes.io/projected/9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b-kube-api-access-8g48b\") pod \"ai-gateway-controller-549c4cbc45-hsm9p\" (UID: \"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b\") " pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.589675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.589657 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b-metrics-certs\") pod \"ai-gateway-controller-549c4cbc45-hsm9p\" (UID: \"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b\") " pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.592000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.591978 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b-metrics-certs\") pod \"ai-gateway-controller-549c4cbc45-hsm9p\" (UID: \"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b\") " pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.643618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.643591 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g48b\" (UniqueName: \"kubernetes.io/projected/9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b-kube-api-access-8g48b\") pod \"ai-gateway-controller-549c4cbc45-hsm9p\" (UID: \"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b\") " pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:00.645409 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:00.645391 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:01.009778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:01.009705 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-549c4cbc45-hsm9p"] Nov 01 12:34:01.012283 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:01.012254 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9d8c16_2bd3_4723_8897_d4fa8c2cb88b.slice/crio-fc11ceab9db23e7973fa90f2ebeb397db008c7729bedff2bd3bfd7a81fe7d194 WatchSource:0}: Error finding container fc11ceab9db23e7973fa90f2ebeb397db008c7729bedff2bd3bfd7a81fe7d194: Status 404 returned error can't find the container with id fc11ceab9db23e7973fa90f2ebeb397db008c7729bedff2bd3bfd7a81fe7d194 Nov 01 12:34:01.048701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:01.048672 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" event={"ID":"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b","Type":"ContainerStarted","Data":"fc11ceab9db23e7973fa90f2ebeb397db008c7729bedff2bd3bfd7a81fe7d194"} Nov 01 12:34:02.053961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:02.053926 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" event={"ID":"9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b","Type":"ContainerStarted","Data":"dc7b236f8472a0273d3b140d0eca89498e2b60985e6c4a30af344b8f9597edfb"} Nov 01 12:34:02.054374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:02.054062 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:02.160768 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:02.160678 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" podStartSLOduration=1.263781953 podStartE2EDuration="2.16066528s" totalImagesPullingTime="896.883327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:02.159068916 +0000 UTC m=+711.909011894" watchObservedRunningTime="2026-11-01 12:34:02.16066528 +0000 UTC m=+711.910608257" Nov 01 12:34:03.521504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:03.521477 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:08.038538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:08.038509 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ph5fh" Nov 01 12:34:09.043264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:09.043237 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qbnms" Nov 01 12:34:13.060637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.060610 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-549c4cbc45-hsm9p" Nov 01 12:34:13.214591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.214562 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-zblbl"] Nov 01 12:34:13.214838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.214780 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" podUID="34aff065-d521-413b-b054-4730e488c301" containerName="manager" containerID="cri-o://31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984" gracePeriod=30 Nov 01 12:34:13.464607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.464583 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:34:13.489234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.489211 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l\" (UniqueName: \"kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l\") pod \"34aff065-d521-413b-b054-4730e488c301\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " Nov 01 12:34:13.489352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.489241 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs\") pod \"34aff065-d521-413b-b054-4730e488c301\" (UID: \"34aff065-d521-413b-b054-4730e488c301\") " Nov 01 12:34:13.491230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.491203 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs" pod "34aff065-d521-413b-b054-4730e488c301" (UID: "34aff065-d521-413b-b054-4730e488c301"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:34:13.491336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.491250 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l" pod "34aff065-d521-413b-b054-4730e488c301" (UID: "34aff065-d521-413b-b054-4730e488c301"). InnerVolumeSpecName "kube-api-access-dtv9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:34:13.590501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.590438 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtv9l\" (UniqueName: \"kubernetes.io/projected/34aff065-d521-413b-b054-4730e488c301-kube-api-access-dtv9l\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:34:13.590501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:13.590463 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34aff065-d521-413b-b054-4730e488c301-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:34:14.106493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.106452 4484 generic.go:356] "Generic (PLEG): container finished" podID="34aff065-d521-413b-b054-4730e488c301" containerID="31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984" exitCode=0 Nov 01 12:34:14.106966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.106504 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" Nov 01 12:34:14.106966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.106510 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" event={"ID":"34aff065-d521-413b-b054-4730e488c301","Type":"ContainerDied","Data":"31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984"} Nov 01 12:34:14.106966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.106619 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-zblbl" event={"ID":"34aff065-d521-413b-b054-4730e488c301","Type":"ContainerDied","Data":"f8397655b976c99aaae2becd8208e975e2f73f2bcf26b272f5542b9f524d7aa3"} Nov 01 12:34:14.106966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.106641 4484 scope.go:122] "RemoveContainer" containerID="31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984" Nov 01 12:34:14.115537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.115518 4484 scope.go:122] "RemoveContainer" containerID="31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984" Nov 01 12:34:14.115850 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:34:14.115829 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984\": container with ID starting with 31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984 not found: ID does not exist" containerID="31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984" Nov 01 12:34:14.115919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.115861 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984"} err="failed to get container status \"31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984\": rpc error: code = NotFound desc = could not find container \"31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984\": container with ID starting with 31d8f63d360515aeca1a71331f7cd63e3bc788fba9f52a744ce731f79c29a984 not found: ID does not exist" Nov 01 12:34:14.257487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.257454 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-zblbl"] Nov 01 12:34:14.303544 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.303519 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-zblbl"] Nov 01 12:34:14.366866 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.366807 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34aff065-d521-413b-b054-4730e488c301" path="/var/lib/kubelet/pods/34aff065-d521-413b-b054-4730e488c301/volumes" Nov 01 12:34:14.724023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:14.723947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:15.221831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:15.221797 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:16.768516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:16.768480 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:16.888012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:16.887976 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:18.221266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:18.221235 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:18.339863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:18.339826 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:30.250727 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:34:30.250690 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:34:41.726258 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:41.726218 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:41.842245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:41.842213 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:42.275261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.275227 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8"] Nov 01 12:34:42.275634 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.275619 4484 cpu_manager.go:414] "containerMap: removing container" podUID="34aff065-d521-413b-b054-4730e488c301" containerName="manager" Nov 01 12:34:42.275634 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.275635 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34aff065-d521-413b-b054-4730e488c301" containerName="manager" Nov 01 12:34:42.275780 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.275708 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="34aff065-d521-413b-b054-4730e488c301" containerName="manager" Nov 01 12:34:42.282273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.282252 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.350776 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.350737 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.350939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.350787 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.350939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.350875 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.350939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.350905 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28x48\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.350939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.350926 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.351092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.351037 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.351137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.351123 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.351205 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.351190 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.351249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.351242 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.353569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.353547 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:42.374199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.374169 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8"] Nov 01 12:34:42.377618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.377595 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ecd640-openshift-default-dockercfg-mfhth\"" Nov 01 12:34:42.452632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452598 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.452837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452669 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.452837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452702 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.452837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452744 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.452837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452783 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.452837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452809 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.452837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452837 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.453155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452866 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x48\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.453155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.452895 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.453788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.453756 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.453983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.453963 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.454104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.454011 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.454773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.454745 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.455071 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.454361 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.456973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.456944 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.457366 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.457347 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.505806 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.505776 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.508632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.508610 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x48\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.595126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.595029 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:42.641730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.641696 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:42.757681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.757648 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:42.902853 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.902766 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr"] Nov 01 12:34:42.907584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.907551 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:42.959931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.959897 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c7206f-openshift-default-dockercfg-8pj7g\"" Nov 01 12:34:42.962796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.962750 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr"] Nov 01 12:34:42.964438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:42.964415 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:43.058898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.058867 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.058974 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66wc\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059050 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059144 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059202 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059266 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059389 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059438 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.059558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.059493 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.062526 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:43.062496 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a20c7db_6279_4745_b955_a3b7459b8b45.slice/crio-c12b5ae494d524f1ab3a50511c776dc3a823bc82ab6c18a8e358ac1d92561463 WatchSource:0}: Error finding container c12b5ae494d524f1ab3a50511c776dc3a823bc82ab6c18a8e358ac1d92561463: Status 404 returned error can't find the container with id c12b5ae494d524f1ab3a50511c776dc3a823bc82ab6c18a8e358ac1d92561463 Nov 01 12:34:43.065760 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.065723 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:43.065955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.065935 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:43.066084 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.066068 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:43.079656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.079623 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:43.082596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.081821 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8"] Nov 01 12:34:43.160886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.160797 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66wc\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.160889 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.160923 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.160955 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.160974 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161025 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161074 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161110 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161152 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161498 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161413 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161498 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161470 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161610 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161519 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161662 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.161974 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.161949 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.163169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.163147 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.163666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.163643 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.214565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.214518 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.214966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.214942 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66wc\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.221692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.221661 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:43.230656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.230624 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" event={"ID":"5a20c7db-6279-4745-b955-a3b7459b8b45","Type":"ContainerStarted","Data":"a9015bd8e959ad1d95309108eec9facd8d55a77ba40c642a666ef0be5330976c"} Nov 01 12:34:43.230774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.230685 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" event={"ID":"5a20c7db-6279-4745-b955-a3b7459b8b45","Type":"ContainerStarted","Data":"c12b5ae494d524f1ab3a50511c776dc3a823bc82ab6c18a8e358ac1d92561463"} Nov 01 12:34:43.265332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.258186 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:43.344842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.344779 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podStartSLOduration=1.344758925 podStartE2EDuration="1.344758925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:43.342552739 +0000 UTC m=+753.092495729" watchObservedRunningTime="2026-11-01 12:34:43.344758925 +0000 UTC m=+753.094701907" Nov 01 12:34:43.370838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.370806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:43.510340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.510292 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:43.543333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.543288 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9"] Nov 01 12:34:43.547846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.547826 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.597283 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.597252 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9514b0-openshift-default-dockercfg-4cdz5\"" Nov 01 12:34:43.598463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.598431 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:43.609053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.609019 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9"] Nov 01 12:34:43.637229 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.637177 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:43.654870 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.654471 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr"] Nov 01 12:34:43.657555 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:43.657516 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b14464_64a5_4ce3_8b8f_65504678c4c8.slice/crio-225e230b2a25f609b20a19fb6960028afe215224654db2b3d0be2b3ce28bdb04 WatchSource:0}: Error finding container 225e230b2a25f609b20a19fb6960028afe215224654db2b3d0be2b3ce28bdb04: Status 404 returned error can't find the container with id 225e230b2a25f609b20a19fb6960028afe215224654db2b3d0be2b3ce28bdb04 Nov 01 12:34:43.660741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.660704 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:43.660846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.660785 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:43.660846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.660824 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:43.670293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670265 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670486 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670415 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670486 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670460 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670560 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670649 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670589 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670656 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwtz\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670806 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.670892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.670866 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.790996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.790904 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.791686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.791658 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.791832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.791814 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.791952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.791935 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.808426 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.808387 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.808656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.808635 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.808795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.808780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.808906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.808889 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.809023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.809009 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwtz\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.809562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.804202 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.818675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.809981 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.818675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.796129 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.818675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.810257 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.818675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.796417 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.818675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.810539 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.818675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.813612 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.848615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.848579 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.861816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.861789 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwtz\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:43.983636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:43.983553 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:44.101569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.101480 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:44.161852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.161814 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:44.239386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.239349 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" event={"ID":"83b14464-64a5-4ce3-8b8f-65504678c4c8","Type":"ContainerStarted","Data":"911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e"} Nov 01 12:34:44.239546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.239414 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" event={"ID":"83b14464-64a5-4ce3-8b8f-65504678c4c8","Type":"ContainerStarted","Data":"225e230b2a25f609b20a19fb6960028afe215224654db2b3d0be2b3ce28bdb04"} Nov 01 12:34:44.348742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.348709 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4"] Nov 01 12:34:44.353479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.353422 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.367062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.367010 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podStartSLOduration=2.366992397 podStartE2EDuration="2.366992397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:44.365172424 +0000 UTC m=+754.115115403" watchObservedRunningTime="2026-11-01 12:34:44.366992397 +0000 UTC m=+754.116935377" Nov 01 12:34:44.403011 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.402978 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:44.404880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.404844 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7427d-openshift-default-dockercfg-zdm4c\"" Nov 01 12:34:44.407167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.407135 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4"] Nov 01 12:34:44.515502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515469 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515526 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515629 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515667 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515697 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2cw\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515823 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515901 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.515981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515966 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.516033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.515997 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.530091 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.530066 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9"] Nov 01 12:34:44.532635 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:44.532606 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a4d823_1015_41d5_87d5_962ea45b4ae7.slice/crio-e6f505f26a3b3e3aecd1c7137c1367a545ddc4377644b8b14918e4c88b673200 WatchSource:0}: Error finding container e6f505f26a3b3e3aecd1c7137c1367a545ddc4377644b8b14918e4c88b673200: Status 404 returned error can't find the container with id e6f505f26a3b3e3aecd1c7137c1367a545ddc4377644b8b14918e4c88b673200 Nov 01 12:34:44.535041 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.535011 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:44.535102 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.535073 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:44.535137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.535105 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:44.560733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.560712 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.616709 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.616771 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.616820 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.616863 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2cw\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.616895 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.616949 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.617005 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.617032 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.617066 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.618671 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.618794 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.619023 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.619058 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.619279 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.620850 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:44.621226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.621058 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.622117 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.622096 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.672907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.672874 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.674095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.674074 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2cw\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:44.971609 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:44.971525 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:45.222653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.222580 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:45.227535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.227501 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:45.245295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.245253 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" event={"ID":"c5a4d823-1015-41d5-87d5-962ea45b4ae7","Type":"ContainerStarted","Data":"b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014"} Nov 01 12:34:45.245451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.245330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" event={"ID":"c5a4d823-1015-41d5-87d5-962ea45b4ae7","Type":"ContainerStarted","Data":"e6f505f26a3b3e3aecd1c7137c1367a545ddc4377644b8b14918e4c88b673200"} Nov 01 12:34:45.245722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.245691 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:45.245821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.245733 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:45.246340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.246294 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:34:45.246428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.246410 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:34:45.333072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.333046 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4"] Nov 01 12:34:45.335919 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:45.335896 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78342bb9_80b3_4763_afdd_210e793e2bbb.slice/crio-f3b4f4e495810a1382121a4d608c212f8a6548d6400c45fe9e556d01cd5e3368 WatchSource:0}: Error finding container f3b4f4e495810a1382121a4d608c212f8a6548d6400c45fe9e556d01cd5e3368: Status 404 returned error can't find the container with id f3b4f4e495810a1382121a4d608c212f8a6548d6400c45fe9e556d01cd5e3368 Nov 01 12:34:45.338453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.338424 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:45.338510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.338483 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:45.338556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.338514 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:34:45.444132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:45.443998 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podStartSLOduration=2.443980324 podStartE2EDuration="2.443980324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:45.442815268 +0000 UTC m=+755.192758246" watchObservedRunningTime="2026-11-01 12:34:45.443980324 +0000 UTC m=+755.193923304" Nov 01 12:34:46.162619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.162583 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:46.167737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.167710 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:46.251912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.251870 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" event={"ID":"78342bb9-80b3-4763-afdd-210e793e2bbb","Type":"ContainerStarted","Data":"fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646"} Nov 01 12:34:46.252095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.251960 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" event={"ID":"78342bb9-80b3-4763-afdd-210e793e2bbb","Type":"ContainerStarted","Data":"f3b4f4e495810a1382121a4d608c212f8a6548d6400c45fe9e556d01cd5e3368"} Nov 01 12:34:46.252155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.252127 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:46.252913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.252893 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:34:46.440491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.440415 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:46.491832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.491787 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podStartSLOduration=2.491772798 podStartE2EDuration="2.491772798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:46.490148025 +0000 UTC m=+756.240091003" watchObservedRunningTime="2026-11-01 12:34:46.491772798 +0000 UTC m=+756.241715817" Nov 01 12:34:46.624184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.624150 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:46.764324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.757120 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:46.972464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.972434 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:46.977290 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:46.977267 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:47.139976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:47.139894 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:47.262364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:47.262331 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:47.263179 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:47.263159 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:34:47.332525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:47.332496 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:47.460863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:47.460787 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:47.986441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:47.986408 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:48.186482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:48.186448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:48.310678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:48.310606 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:48.808822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:48.808791 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:48.979191 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:48.979163 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:49.107394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:49.107292 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:51.081577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.081542 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:51.261642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.261563 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:51.503769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.503739 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892"] Nov 01 12:34:51.507465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.507445 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.612326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.604430 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ecd640-dockercfg-474xw\"" Nov 01 12:34:51.612326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.604501 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ecd640\"" Nov 01 12:34:51.612326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.607752 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892"] Nov 01 12:34:51.634788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.632843 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.634788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.632942 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n262x\" (UniqueName: \"kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x\") pod \"payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.661790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.661760 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75"] Nov 01 12:34:51.665264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.665242 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.718581 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.718543 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75"] Nov 01 12:34:51.734014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.733986 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n262x\" (UniqueName: \"kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x\") pod \"payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.734186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.734046 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8kn\" (UniqueName: \"kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn\") pod \"payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.734186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.734116 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.734273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.734206 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.734864 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.734846 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.787217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.787143 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n262x\" (UniqueName: \"kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x\") pod \"payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.820355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.820324 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:51.836487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.836457 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8kn\" (UniqueName: \"kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn\") pod \"payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.836648 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.836541 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.837855 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.837833 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.890077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.890041 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:51.890252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.890231 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8kn\" (UniqueName: \"kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn\") pod \"payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:51.978067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:51.978032 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:52.075995 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.075918 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:52.184398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.184371 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892"] Nov 01 12:34:52.187560 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:52.187520 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131ac484_20b8_40a4_bc0d_68cc33f01928.slice/crio-5b0de6db88c2b802a9173ecb844e95cd5ddf596f935a65a181be916a2f3f2409 WatchSource:0}: Error finding container 5b0de6db88c2b802a9173ecb844e95cd5ddf596f935a65a181be916a2f3f2409: Status 404 returned error can't find the container with id 5b0de6db88c2b802a9173ecb844e95cd5ddf596f935a65a181be916a2f3f2409 Nov 01 12:34:52.210836 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.210803 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:52.284914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.284884 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" event={"ID":"131ac484-20b8-40a4-bc0d-68cc33f01928","Type":"ContainerStarted","Data":"f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe"} Nov 01 12:34:52.285048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.284942 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" event={"ID":"131ac484-20b8-40a4-bc0d-68cc33f01928","Type":"ContainerStarted","Data":"5b0de6db88c2b802a9173ecb844e95cd5ddf596f935a65a181be916a2f3f2409"} Nov 01 12:34:52.285048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.285017 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:34:52.342424 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.342356 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75"] Nov 01 12:34:52.345939 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:52.345913 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9900f9_e18d_4dc7_a3f1_d7faa0acb339.slice/crio-412d3ba3f9bfcf2c8008c2136ecd211b13a9aea3fe6a6098cc46351a1ba1a16f WatchSource:0}: Error finding container 412d3ba3f9bfcf2c8008c2136ecd211b13a9aea3fe6a6098cc46351a1ba1a16f: Status 404 returned error can't find the container with id 412d3ba3f9bfcf2c8008c2136ecd211b13a9aea3fe6a6098cc46351a1ba1a16f Nov 01 12:34:52.398311 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:52.398263 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" podStartSLOduration=1.398251068 podStartE2EDuration="1.398251068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:52.396583118 +0000 UTC m=+762.146526098" watchObservedRunningTime="2026-11-01 12:34:52.398251068 +0000 UTC m=+762.148194045" Nov 01 12:34:53.190781 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.190740 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:53.302154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.302116 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" event={"ID":"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339","Type":"ContainerStarted","Data":"0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409"} Nov 01 12:34:53.302423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.302177 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" event={"ID":"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339","Type":"ContainerStarted","Data":"412d3ba3f9bfcf2c8008c2136ecd211b13a9aea3fe6a6098cc46351a1ba1a16f"} Nov 01 12:34:53.302423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.302217 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:34:53.356947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.356917 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:53.414026 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.413973 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" podStartSLOduration=2.413960213 podStartE2EDuration="2.413960213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:53.413010324 +0000 UTC m=+763.162953304" watchObservedRunningTime="2026-11-01 12:34:53.413960213 +0000 UTC m=+763.163903185" Nov 01 12:34:53.468520 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.468435 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:53.716397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.716320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:53.893937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:53.893898 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:54.018658 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:54.018628 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:55.884024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:55.883993 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:56.062991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.062936 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:56.459360 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.459326 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr"] Nov 01 12:34:56.462471 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.462443 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.558394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.558365 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-94c94b-dockercfg-8244r\"" Nov 01 12:34:56.558558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.558398 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-94c94b\"" Nov 01 12:34:56.560365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.560344 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr"] Nov 01 12:34:56.600957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.600929 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgds\" (UniqueName: \"kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds\") pod \"payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.601055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.600994 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.702286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.702251 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.702479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.702350 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgds\" (UniqueName: \"kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds\") pod \"payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.702941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.702919 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.754833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.754802 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgds\" (UniqueName: \"kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds\") pod \"payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.775773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.775741 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:56.960050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:56.960018 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:57.140159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.140040 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr"] Nov 01 12:34:57.142871 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:57.142835 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09fa82a_c9e6_4f24_915c_f17b3f3cc969.slice/crio-8bf984804636218a01c42c2d811d723a247b53c5e024ef30e7c39167ce1f820a WatchSource:0}: Error finding container 8bf984804636218a01c42c2d811d723a247b53c5e024ef30e7c39167ce1f820a: Status 404 returned error can't find the container with id 8bf984804636218a01c42c2d811d723a247b53c5e024ef30e7c39167ce1f820a Nov 01 12:34:57.149963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.149943 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:57.293619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.293586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:57.322069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.322038 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" event={"ID":"c09fa82a-c9e6-4f24-915c-f17b3f3cc969","Type":"ContainerStarted","Data":"e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4"} Nov 01 12:34:57.322234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.322094 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" event={"ID":"c09fa82a-c9e6-4f24-915c-f17b3f3cc969","Type":"ContainerStarted","Data":"8bf984804636218a01c42c2d811d723a247b53c5e024ef30e7c39167ce1f820a"} Nov 01 12:34:57.322234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.322162 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:34:57.428962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:57.428857 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" podStartSLOduration=1.4288426379999999 podStartE2EDuration="1.428842638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:57.428179432 +0000 UTC m=+767.178122395" watchObservedRunningTime="2026-11-01 12:34:57.428842638 +0000 UTC m=+767.178785615" Nov 01 12:34:58.342654 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.342623 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:58.567699 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.567672 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:58.681184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.681094 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:58.830552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.830520 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp"] Nov 01 12:34:58.831096 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.831070 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:58.834674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.834652 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:58.897843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.897813 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-c7206f\"" Nov 01 12:34:58.898009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.897916 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-c7206f-dockercfg-z9fjj\"" Nov 01 12:34:58.933961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.933846 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp"] Nov 01 12:34:58.948323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.942946 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m"] Nov 01 12:34:58.948323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.944516 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:58.948323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.944586 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvp4p\" (UniqueName: \"kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p\") pod \"payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:58.965345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.956244 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:58.970987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:58.970959 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:34:59.004319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.003820 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m"] Nov 01 12:34:59.046184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.046153 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:59.046380 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.046203 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvp4p\" (UniqueName: \"kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p\") pod \"payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:59.046380 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.046248 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wb8\" (UniqueName: \"kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8\") pod \"payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.046380 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.046278 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.046959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.046939 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:59.099988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.099954 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvp4p\" (UniqueName: \"kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p\") pod \"payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:59.147666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.147627 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wb8\" (UniqueName: \"kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8\") pod \"payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.147828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.147686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.148401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.148380 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.148463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.148406 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:34:59.204262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.204232 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wb8\" (UniqueName: \"kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8\") pod \"payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.284185 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.284156 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:34:59.513252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.513219 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp"] Nov 01 12:34:59.516282 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:59.516247 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3708e19_6de2_4ba2_a1c7_23e010ab81da.slice/crio-2c6e1677bd897f1eb8236d35cd0602bd11298a5df117937fbbba79041dfc6534 WatchSource:0}: Error finding container 2c6e1677bd897f1eb8236d35cd0602bd11298a5df117937fbbba79041dfc6534: Status 404 returned error can't find the container with id 2c6e1677bd897f1eb8236d35cd0602bd11298a5df117937fbbba79041dfc6534 Nov 01 12:34:59.647272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:34:59.647243 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m"] Nov 01 12:34:59.649557 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:34:59.649522 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e820e6b_7262_4c30_acec_14251eaf26ed.slice/crio-dcea8066255c47c719c40dd4f76c90e490b2a92af08f82f0a96a06f5ab322157 WatchSource:0}: Error finding container dcea8066255c47c719c40dd4f76c90e490b2a92af08f82f0a96a06f5ab322157: Status 404 returned error can't find the container with id dcea8066255c47c719c40dd4f76c90e490b2a92af08f82f0a96a06f5ab322157 Nov 01 12:35:00.251114 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:35:00.251079 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:35:00.337331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.337270 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" event={"ID":"e3708e19-6de2-4ba2-a1c7-23e010ab81da","Type":"ContainerStarted","Data":"85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb"} Nov 01 12:35:00.337518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.337464 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:35:00.337518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.337497 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" event={"ID":"e3708e19-6de2-4ba2-a1c7-23e010ab81da","Type":"ContainerStarted","Data":"2c6e1677bd897f1eb8236d35cd0602bd11298a5df117937fbbba79041dfc6534"} Nov 01 12:35:00.338798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.338773 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" event={"ID":"8e820e6b-7262-4c30-acec-14251eaf26ed","Type":"ContainerStarted","Data":"67e12cef59740c385be2f1179a70ef4013afdef6b7f7eaaf83aaad4046338c00"} Nov 01 12:35:00.338890 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.338814 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" event={"ID":"8e820e6b-7262-4c30-acec-14251eaf26ed","Type":"ContainerStarted","Data":"dcea8066255c47c719c40dd4f76c90e490b2a92af08f82f0a96a06f5ab322157"} Nov 01 12:35:00.338890 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.338844 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:35:00.442959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.442906 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" podStartSLOduration=2.442891817 podStartE2EDuration="2.442891817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:00.441201321 +0000 UTC m=+770.191144299" watchObservedRunningTime="2026-11-01 12:35:00.442891817 +0000 UTC m=+770.192834794" Nov 01 12:35:00.598127 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:00.598020 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" podStartSLOduration=2.598005232 podStartE2EDuration="2.598005232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:00.597146078 +0000 UTC m=+770.347089057" watchObservedRunningTime="2026-11-01 12:35:00.598005232 +0000 UTC m=+770.347948210" Nov 01 12:35:01.114940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:01.114908 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:35:03.186745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.186712 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:03.309446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.309406 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 12:35:03.364074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.364045 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:03.770356 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.770313 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs"] Nov 01 12:35:03.774503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.774484 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:03.824448 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.824422 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-06fd6b\"" Nov 01 12:35:03.851796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.851764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:03.870222 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.870195 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-06fd6b-dockercfg-pzj2g\"" Nov 01 12:35:03.872104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.872084 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs"] Nov 01 12:35:03.911534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.911501 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:03.911691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:03.911663 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krmp\" (UniqueName: \"kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp\") pod \"payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:04.012562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.012509 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:04.012743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.012652 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krmp\" (UniqueName: \"kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp\") pod \"payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:04.013219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.013199 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:04.063640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.063534 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:04.065116 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.065096 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krmp\" (UniqueName: \"kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp\") pod \"payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:04.088096 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.088067 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:04.224579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.224541 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:04.310577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.310544 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 12:35:04.339818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.339715 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:04.466470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:04.466331 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs"] Nov 01 12:35:04.469242 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:35:04.469211 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99915904_ffac_435d_a0a5_da93549daca6.slice/crio-1f48be52baa9de1dae9c4f117db5d870838c5aad1df10d09d41de45a4350ba4f WatchSource:0}: Error finding container 1f48be52baa9de1dae9c4f117db5d870838c5aad1df10d09d41de45a4350ba4f: Status 404 returned error can't find the container with id 1f48be52baa9de1dae9c4f117db5d870838c5aad1df10d09d41de45a4350ba4f Nov 01 12:35:05.379375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:05.379345 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" event={"ID":"99915904-ffac-435d-a0a5-da93549daca6","Type":"ContainerStarted","Data":"9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c"} Nov 01 12:35:05.379769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:05.379394 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" event={"ID":"99915904-ffac-435d-a0a5-da93549daca6","Type":"ContainerStarted","Data":"1f48be52baa9de1dae9c4f117db5d870838c5aad1df10d09d41de45a4350ba4f"} Nov 01 12:35:05.379769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:05.379494 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:05.485608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:05.485555 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" podStartSLOduration=2.485539802 podStartE2EDuration="2.485539802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:05.484677476 +0000 UTC m=+775.234620455" watchObservedRunningTime="2026-11-01 12:35:05.485539802 +0000 UTC m=+775.235482780" Nov 01 12:35:05.679545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:05.679452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:05.850762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:05.850726 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:08.330736 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:08.330710 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:35:09.206745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.206700 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl"] Nov 01 12:35:09.210514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.210491 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.261448 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.261417 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-06fd6b-serving-cert\"" Nov 01 12:35:09.261608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.261588 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-758k9\"" Nov 01 12:35:09.261722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.261706 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 12:35:09.261903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.261890 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 12:35:09.262061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.261969 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 12:35:09.264376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.264349 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl"] Nov 01 12:35:09.294835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.294807 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.294963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.294841 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkt8\" (UniqueName: \"kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.294963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.294889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.295098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.295074 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.405992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.405955 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.406580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.406001 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkt8\" (UniqueName: \"kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.406580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.406075 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.406580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.406143 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.406580 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:35:09.406361 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-serving-cert: secret "maas-api-e2e-worker-w4-06fd6b-serving-cert" not found Nov 01 12:35:09.406580 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:35:09.406429 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls podName:27901c66-1863-4fa4-8888-ec5e7fbb2782 nodeName:}" failed. No retries permitted until 2026-11-01 12:35:09.906409271 +0000 UTC m=+779.656352241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls") pod "maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" (UID: "27901c66-1863-4fa4-8888-ec5e7fbb2782") : secret "maas-api-e2e-worker-w4-06fd6b-serving-cert" not found Nov 01 12:35:09.411525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.411288 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.414661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.414629 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:09.415314 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.415280 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.465946 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.465873 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkt8\" (UniqueName: \"kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.561278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.561245 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:35:09.592445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.592415 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:09.711871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.711771 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:09.911171 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.911137 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:09.913684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:09.913652 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\") pod \"maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:10.122743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:10.122707 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:10.479976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:10.479945 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:10.514798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:10.514765 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl"] Nov 01 12:35:10.518146 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:35:10.518116 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27901c66_1863_4fa4_8888_ec5e7fbb2782.slice/crio-12887eef515e23eb72654b0df0f109751b5688d8782ef8474bd47c6040435e57 WatchSource:0}: Error finding container 12887eef515e23eb72654b0df0f109751b5688d8782ef8474bd47c6040435e57: Status 404 returned error can't find the container with id 12887eef515e23eb72654b0df0f109751b5688d8782ef8474bd47c6040435e57 Nov 01 12:35:10.694749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:10.694711 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:11.367410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:11.367380 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:35:11.367854 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:11.367828 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:35:11.427272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:11.427234 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" event={"ID":"27901c66-1863-4fa4-8888-ec5e7fbb2782","Type":"ContainerStarted","Data":"12887eef515e23eb72654b0df0f109751b5688d8782ef8474bd47c6040435e57"} Nov 01 12:35:12.433537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:12.433492 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" event={"ID":"27901c66-1863-4fa4-8888-ec5e7fbb2782","Type":"ContainerStarted","Data":"8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335"} Nov 01 12:35:12.433933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:12.433557 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:12.544408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:12.542841 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" podStartSLOduration=2.565560504 podStartE2EDuration="3.542821632s" totalImagesPullingTime="977.261128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:12.541510192 +0000 UTC m=+782.291453172" watchObservedRunningTime="2026-11-01 12:35:12.542821632 +0000 UTC m=+782.292764611" Nov 01 12:35:13.067900 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.067859 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:13.227042 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.226907 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:13.477551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.477512 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj"] Nov 01 12:35:13.481250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.481230 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.533702 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.533672 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj"] Nov 01 12:35:13.588330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.587682 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5b1bd4\"" Nov 01 12:35:13.588330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.587831 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5b1bd4-dockercfg-rpvbv\"" Nov 01 12:35:13.610217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.609069 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.610217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.609139 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zx6\" (UniqueName: \"kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6\") pod \"payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.637245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.637196 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz"] Nov 01 12:35:13.669737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.669704 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.697377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.697341 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz"] Nov 01 12:35:13.710493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.710461 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.710677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.710501 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zx6\" (UniqueName: \"kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6\") pod \"payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.711208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.711183 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.769029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.768952 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zx6\" (UniqueName: \"kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6\") pod \"payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.793613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.793585 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:13.811094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.811063 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69sqf\" (UniqueName: \"kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf\") pod \"payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.811211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.811132 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.913432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.912243 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69sqf\" (UniqueName: \"kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf\") pod \"payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.913432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.912357 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.913432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.913373 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.968576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.968518 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69sqf\" (UniqueName: \"kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf\") pod \"payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:13.985500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:13.985474 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:14.183002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.182916 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj"] Nov 01 12:35:14.186031 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:35:14.186000 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f666c7_0692_4a6b_bb77_7d8bdda8e342.slice/crio-e241db8c5a59a5d2d9dc0f3e2d68c1a8fd41164e5879d584d4a630f3df7d5241 WatchSource:0}: Error finding container e241db8c5a59a5d2d9dc0f3e2d68c1a8fd41164e5879d584d4a630f3df7d5241: Status 404 returned error can't find the container with id e241db8c5a59a5d2d9dc0f3e2d68c1a8fd41164e5879d584d4a630f3df7d5241 Nov 01 12:35:14.381534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.381379 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz"] Nov 01 12:35:14.383745 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:35:14.383717 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d7c973_1fdd_44ed_92a1_24651716dfef.slice/crio-18d5f072f0626548e80a72e68edc08d63c94babbf6220f2d141375c76d338864 WatchSource:0}: Error finding container 18d5f072f0626548e80a72e68edc08d63c94babbf6220f2d141375c76d338864: Status 404 returned error can't find the container with id 18d5f072f0626548e80a72e68edc08d63c94babbf6220f2d141375c76d338864 Nov 01 12:35:14.467952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.458502 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" event={"ID":"24d7c973-1fdd-44ed-92a1-24651716dfef","Type":"ContainerStarted","Data":"950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283"} Nov 01 12:35:14.467952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.458567 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" event={"ID":"24d7c973-1fdd-44ed-92a1-24651716dfef","Type":"ContainerStarted","Data":"18d5f072f0626548e80a72e68edc08d63c94babbf6220f2d141375c76d338864"} Nov 01 12:35:14.467952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.459466 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:14.469385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.469354 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" event={"ID":"80f666c7-0692-4a6b-bb77-7d8bdda8e342","Type":"ContainerStarted","Data":"cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba"} Nov 01 12:35:14.469518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.469400 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" event={"ID":"80f666c7-0692-4a6b-bb77-7d8bdda8e342","Type":"ContainerStarted","Data":"e241db8c5a59a5d2d9dc0f3e2d68c1a8fd41164e5879d584d4a630f3df7d5241"} Nov 01 12:35:14.469518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.469459 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:14.572347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.572267 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" podStartSLOduration=1.572247505 podStartE2EDuration="1.572247505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:14.569149632 +0000 UTC m=+784.319092610" watchObservedRunningTime="2026-11-01 12:35:14.572247505 +0000 UTC m=+784.322190484" Nov 01 12:35:14.961525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:14.961491 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:16.386864 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:16.386837 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 12:35:16.493778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:16.493710 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" podStartSLOduration=3.493691666 podStartE2EDuration="3.493691666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:14.681217376 +0000 UTC m=+784.431160355" watchObservedRunningTime="2026-11-01 12:35:16.493691666 +0000 UTC m=+786.243634645" Nov 01 12:35:18.443587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:18.443556 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 12:35:18.753900 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:18.753868 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:35:19.496042 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.496012 4484 generic.go:356] "Generic (PLEG): container finished" podID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerID="911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e" exitCode=0 Nov 01 12:35:19.496547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.496100 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" event={"ID":"83b14464-64a5-4ce3-8b8f-65504678c4c8","Type":"ContainerDied","Data":"911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e"} Nov 01 12:35:19.496623 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.496565 4484 scope.go:122] "RemoveContainer" containerID="911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e" Nov 01 12:35:19.497859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.497831 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:19.497943 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.497893 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:19.497943 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.497926 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:19.919348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:19.919247 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:20.040133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:20.040096 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:20.165201 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:20.165161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:20.502655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:20.502624 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" event={"ID":"83b14464-64a5-4ce3-8b8f-65504678c4c8","Type":"ContainerStarted","Data":"cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04"} Nov 01 12:35:21.223425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:21.223393 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:35:21.223716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:21.223693 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 12:35:21.223777 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:21.223746 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 12:35:22.222891 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:22.222852 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 12:35:22.223395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:22.222913 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 12:35:23.222511 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:23.222475 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 12:35:23.222692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:23.222530 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 12:35:24.222869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:24.222837 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 12:35:24.223256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:24.222906 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 12:35:24.930526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:24.930492 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:25.222538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:25.222445 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 12:35:25.222538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:25.222518 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 12:35:25.477126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:25.477099 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 12:35:26.226802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:26.226774 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:35:26.227197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:26.227171 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:35:26.227760 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:26.227742 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:35:26.482044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:26.481955 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 12:35:28.750025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.749991 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv"] Nov 01 12:35:28.753689 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.753672 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.803896 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.803854 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcj2b\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.803936 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.803972 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.804049 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.804078 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.804132 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.804158 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.804202 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.804465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.804223 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.805534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.805501 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv"] Nov 01 12:35:28.844345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.844288 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:35:28.858168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.858143 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-06fd6b-openshift-default-dockercfg-6x29r\"" Nov 01 12:35:28.904977 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.904945 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.904981 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905018 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905042 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905090 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcj2b\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905139 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905174 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905397 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905497 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905599 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905609 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.905936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905908 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.906044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905652 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.906044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.905853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.907585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.907551 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.908251 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.908234 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.963892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.963863 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcj2b\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:28.965183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:28.965165 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:29.071490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.071405 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:29.491635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.491608 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv"] Nov 01 12:35:29.496092 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:35:29.495547 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1d5528_6aba_46ee_b3da_a7509da518b9.slice/crio-c6f69219227d87130f2e09294a75b3e5b4dd048f02f00921ae6cc1449d539254 WatchSource:0}: Error finding container c6f69219227d87130f2e09294a75b3e5b4dd048f02f00921ae6cc1449d539254: Status 404 returned error can't find the container with id c6f69219227d87130f2e09294a75b3e5b4dd048f02f00921ae6cc1449d539254 Nov 01 12:35:29.498859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.498826 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:29.498933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.498890 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:29.498933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.498922 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:29.546112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.546087 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" event={"ID":"9a1d5528-6aba-46ee-b3da-a7509da518b9","Type":"ContainerStarted","Data":"c6f69219227d87130f2e09294a75b3e5b4dd048f02f00921ae6cc1449d539254"} Nov 01 12:35:29.777679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.777598 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:29.942653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:29.942625 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:30.095427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:30.083709 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:30.296506 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:35:30.296449 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:35:30.588280 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:30.583462 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" event={"ID":"9a1d5528-6aba-46ee-b3da-a7509da518b9","Type":"ContainerStarted","Data":"8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186"} Nov 01 12:35:30.793326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:30.792946 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podStartSLOduration=2.7929235439999998 podStartE2EDuration="2.792923544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:30.741590212 +0000 UTC m=+800.491533191" watchObservedRunningTime="2026-11-01 12:35:30.792923544 +0000 UTC m=+800.542866524" Nov 01 12:35:31.076326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:31.075662 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:31.080323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:31.079418 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 12:35:31.080323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:31.079477 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 12:35:31.765169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:31.765137 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:32.076387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:32.075747 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 12:35:32.076387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:32.075809 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 12:35:32.380430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:32.380275 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:32.496292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:32.496260 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:33.080266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:33.079805 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 12:35:33.080266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:33.079870 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 12:35:34.078326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:34.075752 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 12:35:34.078326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:34.075815 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 12:35:35.077326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:35.077190 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 12:35:35.077326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:35.077268 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 12:35:35.563128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:35.563087 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:36.076745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.076714 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:36.077127 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.077106 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:36.077929 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.077913 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 12:35:36.313998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.313961 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr"] Nov 01 12:35:36.319800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.319771 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.370841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.370791 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"kube-root-ca.crt\"" Nov 01 12:35:36.371098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.371070 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"default-dockercfg-8xv8p\"" Nov 01 12:35:36.372139 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.372104 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"openshift-service-ca.crt\"" Nov 01 12:35:36.372642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.372610 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"simulator-w0-94c94b-kserve-self-signed-certs\"" Nov 01 12:35:36.373572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.373550 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr"] Nov 01 12:35:36.399435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.399361 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:36.442519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.442487 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhb9\" (UniqueName: \"kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.442519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.442531 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.442755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.442569 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.442755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.442736 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.442851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.442808 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.442909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.442888 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.545676 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.545739 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.545793 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhb9\" (UniqueName: \"kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.545826 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.545864 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.545997 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.546671 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.547578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.546927 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.551721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.551692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.566241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.566176 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.577879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.577754 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.599580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.599542 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhb9\" (UniqueName: \"kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9\") pod \"simulator-w0-94c94b-kserve-fc68b7f78-lfgkr\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:36.634636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:36.634608 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:37.009406 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:37.009380 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr"] Nov 01 12:35:37.012195 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:35:37.012155 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode52f59e0_1df0_41f0_b363_bb5962bd92a1.slice/crio-9928ac030ab329c79c65265ca9c99d4009c0ecc6ec47976c40c8d06316df5c08 WatchSource:0}: Error finding container 9928ac030ab329c79c65265ca9c99d4009c0ecc6ec47976c40c8d06316df5c08: Status 404 returned error can't find the container with id 9928ac030ab329c79c65265ca9c99d4009c0ecc6ec47976c40c8d06316df5c08 Nov 01 12:35:37.645167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:37.645126 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" event={"ID":"e52f59e0-1df0-41f0-b363-bb5962bd92a1","Type":"ContainerStarted","Data":"3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d"} Nov 01 12:35:37.645584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:37.645178 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" event={"ID":"e52f59e0-1df0-41f0-b363-bb5962bd92a1","Type":"ContainerStarted","Data":"9928ac030ab329c79c65265ca9c99d4009c0ecc6ec47976c40c8d06316df5c08"} Nov 01 12:35:37.751925 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:37.751873 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podStartSLOduration=1.751859185 podStartE2EDuration="1.751859185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:37.749860592 +0000 UTC m=+807.499803570" watchObservedRunningTime="2026-11-01 12:35:37.751859185 +0000 UTC m=+807.501802163" Nov 01 12:35:38.635692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:38.635647 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:39.656614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:39.656584 4484 generic.go:356] "Generic (PLEG): container finished" podID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerID="a9015bd8e959ad1d95309108eec9facd8d55a77ba40c642a666ef0be5330976c" exitCode=0 Nov 01 12:35:39.657049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:39.656663 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" event={"ID":"5a20c7db-6279-4745-b955-a3b7459b8b45","Type":"ContainerDied","Data":"a9015bd8e959ad1d95309108eec9facd8d55a77ba40c642a666ef0be5330976c"} Nov 01 12:35:39.657167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:39.657153 4484 scope.go:122] "RemoveContainer" containerID="a9015bd8e959ad1d95309108eec9facd8d55a77ba40c642a666ef0be5330976c" Nov 01 12:35:39.658336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:39.658290 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:39.658398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:39.658376 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:39.658435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:39.658407 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:40.237322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.235488 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:35:40.442766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.442728 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg"] Nov 01 12:35:40.470112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.470083 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.509376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.500246 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg"] Nov 01 12:35:40.537624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.537591 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-f7427d-serving-cert\"" Nov 01 12:35:40.624980 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.624947 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.625184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.625154 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.625287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.625266 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.625360 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.625315 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpsvm\" (UniqueName: \"kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.640009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.639979 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:40.676391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.676354 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" event={"ID":"5a20c7db-6279-4745-b955-a3b7459b8b45","Type":"ContainerStarted","Data":"8fe60b503810a4c0b1acb8077a31abc1e579938039720f4a31e39e2392d936c8"} Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.727180 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.727317 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.727358 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpsvm\" (UniqueName: \"kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.727415 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.728773 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:35:40.728876 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-serving-cert: secret "maas-api-e2e-worker-w6-f7427d-serving-cert" not found Nov 01 12:35:40.735359 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:35:40.728933 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls podName:5a4ab1d3-6b74-483a-9602-03100e07b77b nodeName:}" failed. No retries permitted until 2026-11-01 12:35:41.228914203 +0000 UTC m=+810.978857166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls") pod "maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" (UID: "5a4ab1d3-6b74-483a-9602-03100e07b77b") : secret "maas-api-e2e-worker-w6-f7427d-serving-cert" not found Nov 01 12:35:40.748322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.744878 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.800777 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.800692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpsvm\" (UniqueName: \"kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:40.812417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:40.812379 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:41.133153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.133078 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:41.232025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.231989 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:41.234408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.234384 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:41.383379 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.382497 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:41.597481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.597440 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:35:41.598361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.598332 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 12:35:41.598479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.598388 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 12:35:41.763325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:41.756234 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg"] Nov 01 12:35:42.596585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:42.596546 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 12:35:42.596822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:42.596603 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 12:35:42.704427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:42.704398 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:42.704981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:42.704956 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" event={"ID":"5a4ab1d3-6b74-483a-9602-03100e07b77b","Type":"ContainerStarted","Data":"5d4606e636846b885bee518082c7813355215a84ce27724bbe0df3dd0425b313"} Nov 01 12:35:43.273346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.273166 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:43.401396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.401351 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:43.529103 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.528997 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:43.596637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.596596 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 12:35:43.596820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.596656 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 12:35:43.711987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.711945 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" event={"ID":"5a4ab1d3-6b74-483a-9602-03100e07b77b","Type":"ContainerStarted","Data":"3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51"} Nov 01 12:35:43.712167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:43.712009 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:44.596743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:44.596704 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 12:35:44.597162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:44.596760 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 12:35:45.250324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:45.249390 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:35:45.596598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:45.596509 4484 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 12:35:45.596598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:45.596568 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 12:35:46.077518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.077476 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:46.600650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.600618 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:35:46.601767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.601747 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 12:35:46.635833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.635811 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:46.666834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.666807 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:46.739019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.738993 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:35:46.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:46.801928 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" podStartSLOduration=5.867309614 podStartE2EDuration="6.801915834s" totalImagesPullingTime="934.60622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:43.831724874 +0000 UTC m=+813.581667852" watchObservedRunningTime="2026-11-01 12:35:46.801915834 +0000 UTC m=+816.551858811" Nov 01 12:35:49.722174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:49.722147 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 12:35:50.208919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.208856 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:35:50.752385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.752352 4484 generic.go:356] "Generic (PLEG): container finished" podID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerID="b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014" exitCode=0 Nov 01 12:35:50.752784 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.752422 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" event={"ID":"c5a4d823-1015-41d5-87d5-962ea45b4ae7","Type":"ContainerDied","Data":"b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014"} Nov 01 12:35:50.752832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.752797 4484 scope.go:122] "RemoveContainer" containerID="b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014" Nov 01 12:35:50.754027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.753996 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:50.754108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.754061 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:50.754108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:50.754092 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:51.058030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:51.057887 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:51.794575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:51.794541 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" event={"ID":"c5a4d823-1015-41d5-87d5-962ea45b4ae7","Type":"ContainerStarted","Data":"7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1"} Nov 01 12:35:52.162759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:52.162669 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:35:52.163086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:52.163061 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 12:35:52.163249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:52.163221 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 12:35:53.162834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:53.162797 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 12:35:53.163491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:53.163443 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 12:35:54.165805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:54.165760 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 12:35:54.166353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:54.165856 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 12:35:54.197937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:54.197905 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:54.772354 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:54.772317 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:54.893718 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:54.893659 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:55.020989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:55.020962 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:55.162736 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:55.162644 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 12:35:55.163024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:55.162983 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 12:35:56.165721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:56.165685 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 12:35:56.166391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:56.166341 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 12:35:56.434855 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:56.434766 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:57.166513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:57.166485 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:35:57.166872 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:57.166755 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:35:57.167603 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:57.167582 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 12:35:58.436915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:58.436881 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:58.558329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:58.558175 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:59.008451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.008410 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:59.129129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.129087 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:59.246531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.246493 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:35:59.859082 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.859049 4484 generic.go:356] "Generic (PLEG): container finished" podID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerID="fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646" exitCode=0 Nov 01 12:35:59.859441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.859118 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" event={"ID":"78342bb9-80b3-4763-afdd-210e793e2bbb","Type":"ContainerDied","Data":"fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646"} Nov 01 12:35:59.859484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.859473 4484 scope.go:122] "RemoveContainer" containerID="fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646" Nov 01 12:35:59.860795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.860768 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:59.860863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.860841 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:59.860939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.860873 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:35:59.992610 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.992580 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh"] Nov 01 12:35:59.997221 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:35:59.997194 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078253 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078327 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078528 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078555 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdtb\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078584 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078639 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078663 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.078914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.078759 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.120317 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.120260 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh"] Nov 01 12:36:00.180250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180219 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180260 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180280 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdtb\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180316 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180355 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180373 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180723 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180461 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180723 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180697 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180716 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180730 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.180829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.180798 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.181135 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.181110 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.181265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.181169 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.181265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.181186 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.183192 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.183171 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.183557 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.183528 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.233276 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.233244 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.233612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.233585 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdtb\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.252120 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:36:00.252072 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:36:00.315060 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.313490 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:00.835456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.835432 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh"] Nov 01 12:36:00.838265 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:36:00.838231 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6e017e_35a9_4f5c_a735_fc801396a2bb.slice/crio-dac2b23424ba6673b0da0f90b137f6cfbc3728128b598554fd4d4661a5907baa WatchSource:0}: Error finding container dac2b23424ba6673b0da0f90b137f6cfbc3728128b598554fd4d4661a5907baa: Status 404 returned error can't find the container with id dac2b23424ba6673b0da0f90b137f6cfbc3728128b598554fd4d4661a5907baa Nov 01 12:36:00.841142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.841109 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:36:00.841217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.841172 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:36:00.841217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.841203 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:36:00.871799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.871762 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" event={"ID":"78342bb9-80b3-4763-afdd-210e793e2bbb","Type":"ContainerStarted","Data":"0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5"} Nov 01 12:36:00.873128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.873094 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" event={"ID":"0d6e017e-35a9-4f5c-a735-fc801396a2bb","Type":"ContainerStarted","Data":"dac2b23424ba6673b0da0f90b137f6cfbc3728128b598554fd4d4661a5907baa"} Nov 01 12:36:00.972782 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.972707 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:36:00.973581 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.973551 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 12:36:00.973794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:00.973603 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 12:36:01.870336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:01.870095 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:01.892070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:01.891465 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" event={"ID":"0d6e017e-35a9-4f5c-a735-fc801396a2bb","Type":"ContainerStarted","Data":"cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f"} Nov 01 12:36:01.974324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:01.973861 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 12:36:01.974324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:01.973945 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 12:36:02.262521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:02.262408 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:36:02.314072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:02.314040 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:02.315697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:02.315432 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 12:36:02.315697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:02.315476 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 12:36:02.984326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:02.982808 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 12:36:02.984326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:02.982896 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 12:36:03.314664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:03.314578 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 12:36:03.314664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:03.314631 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 12:36:03.700068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:03.699964 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podStartSLOduration=4.69995017 podStartE2EDuration="4.69995017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:02.017969819 +0000 UTC m=+831.767912798" watchObservedRunningTime="2026-11-01 12:36:03.69995017 +0000 UTC m=+833.449893149" Nov 01 12:36:03.701385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:03.701365 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:36:03.975774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:03.975679 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 12:36:03.976043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:03.976012 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 12:36:04.263756 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:04.263722 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:04.314468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:04.314425 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 12:36:04.314635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:04.314501 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 12:36:04.973089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:04.972648 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 12:36:04.973089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:04.972723 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 12:36:05.314564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:05.314530 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 12:36:05.314962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:05.314586 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 12:36:06.018602 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:06.018562 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:36:06.023073 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:06.023048 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:36:06.315824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:06.314546 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 12:36:06.315824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:06.314606 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 12:36:07.109885 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.109848 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:07.318358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.318332 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:07.318818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.318717 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:07.319408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.319386 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:36:07.535385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.535348 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj"] Nov 01 12:36:07.540215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.540193 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.588802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.588774 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"openshift-service-ca.crt\"" Nov 01 12:36:07.588802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.588792 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"kube-root-ca.crt\"" Nov 01 12:36:07.589003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.588920 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"default-dockercfg-vp2p9\"" Nov 01 12:36:07.591721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.591700 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"simulator-w1-5b1bd4-kserve-self-signed-certs\"" Nov 01 12:36:07.612849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.612807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:07.614579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.614558 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj"] Nov 01 12:36:07.623584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.623562 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.623679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.623603 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.623679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.623648 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.623896 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.623871 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.623934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.623918 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq2m\" (UniqueName: \"kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.624015 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.624002 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.725970 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.726054 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.726226 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.726259 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq2m\" (UniqueName: \"kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.726340 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.726397 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.726888 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.727563 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.733333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.727774 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.756323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.750359 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.759001 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.758958 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.780734 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.780700 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq2m\" (UniqueName: \"kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m\") pod \"simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:07.852386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:07.852321 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:08.216086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:08.216063 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj"] Nov 01 12:36:08.218487 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:36:08.218452 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa169038_e9bd_47c5_b6dc_093f8a1d9c08.slice/crio-2d673411835841fe001b6e18cc6cc2f64198daa15d0e804822a31e819c023d32 WatchSource:0}: Error finding container 2d673411835841fe001b6e18cc6cc2f64198daa15d0e804822a31e819c023d32: Status 404 returned error can't find the container with id 2d673411835841fe001b6e18cc6cc2f64198daa15d0e804822a31e819c023d32 Nov 01 12:36:08.937936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:08.937898 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" event={"ID":"aa169038-e9bd-47c5-b6dc-093f8a1d9c08","Type":"ContainerStarted","Data":"443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c"} Nov 01 12:36:08.937936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:08.937941 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" event={"ID":"aa169038-e9bd-47c5-b6dc-093f8a1d9c08","Type":"ContainerStarted","Data":"2d673411835841fe001b6e18cc6cc2f64198daa15d0e804822a31e819c023d32"} Nov 01 12:36:09.044942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:09.044893 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podStartSLOduration=2.044879213 podStartE2EDuration="2.044879213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:09.042750381 +0000 UTC m=+838.792693360" watchObservedRunningTime="2026-11-01 12:36:09.044879213 +0000 UTC m=+838.794822190" Nov 01 12:36:09.852952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:09.852913 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:12.355538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:12.355505 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:12.777025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:12.776983 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:13.336089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:13.336058 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:13.474935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:13.474909 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:13.701707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:13.701622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:17.522525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:17.522477 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:17.853347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:17.853256 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:17.865528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:17.865507 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:17.995089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:17.995065 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:36:19.665518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:19.665483 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:20.226445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:20.226409 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:20.365715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:20.365678 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:20.483640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:20.483536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:22.738717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:22.738678 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:22.959332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:22.959286 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:36:24.375699 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:24.375616 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:27.662888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:27.662849 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:29.895504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:29.895472 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:30.251341 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:36:30.251284 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:36:31.154533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:31.154505 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:31.708134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:31.708085 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:31.835797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:31.835764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:31.954900 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:31.954872 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:33.051900 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:33.051864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:37.320545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.320510 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf"] Nov 01 12:36:37.324691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.324668 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.398163 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.398134 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:37.420549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.420526 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"default-dockercfg-wj74h\"" Nov 01 12:36:37.420549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.420531 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"kube-root-ca.crt\"" Nov 01 12:36:37.420720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.420587 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"openshift-service-ca.crt\"" Nov 01 12:36:37.420720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.420663 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"simulator-w3-ecd640-kserve-self-signed-certs\"" Nov 01 12:36:37.422961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.422944 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf"] Nov 01 12:36:37.432088 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.432064 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.432176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.432111 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.432176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.432133 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.432248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.432222 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdqg\" (UniqueName: \"kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.432323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.432297 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.432366 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.432355 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533297 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533395 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533495 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533549 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533582 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533621 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdqg\" (UniqueName: \"kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.533955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.533927 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.534159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.534017 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.534258 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.534067 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.535688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.535663 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.535935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.535919 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.585917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.585863 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdqg\" (UniqueName: \"kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg\") pod \"simulator-w3-ecd640-kserve-8478c6d8cb-b8czf\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.637130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.637103 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:37.679717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.679686 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:36:37.946359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:37.946266 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:38.003784 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:38.003761 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf"] Nov 01 12:36:38.005772 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:36:38.005740 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7adbc4ea_a43d_4441_8cda_9fed508bfbda.slice/crio-8aeb8c6bb1a110cd282e72b067461f2e3a64cfaf27376d2e5df85f3cc1eeff0a WatchSource:0}: Error finding container 8aeb8c6bb1a110cd282e72b067461f2e3a64cfaf27376d2e5df85f3cc1eeff0a: Status 404 returned error can't find the container with id 8aeb8c6bb1a110cd282e72b067461f2e3a64cfaf27376d2e5df85f3cc1eeff0a Nov 01 12:36:38.094656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:38.094426 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" event={"ID":"7adbc4ea-a43d-4441-8cda-9fed508bfbda","Type":"ContainerStarted","Data":"8aeb8c6bb1a110cd282e72b067461f2e3a64cfaf27376d2e5df85f3cc1eeff0a"} Nov 01 12:36:39.101063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:39.101031 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" event={"ID":"7adbc4ea-a43d-4441-8cda-9fed508bfbda","Type":"ContainerStarted","Data":"6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511"} Nov 01 12:36:39.207970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:39.207923 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podStartSLOduration=2.207909474 podStartE2EDuration="2.207909474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:39.206501304 +0000 UTC m=+868.956444282" watchObservedRunningTime="2026-11-01 12:36:39.207909474 +0000 UTC m=+868.957852452" Nov 01 12:36:39.637908 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:39.637880 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:42.833778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:42.833730 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:47.637817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:47.637777 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:47.650361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:47.650339 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:47.736538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:47.736506 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:48.165061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:48.165028 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:36:52.662861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:52.662818 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:53.001428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:53.001400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:36:53.060604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:53.060578 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:36:57.577425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:36:57.577372 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:00.252546 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:37:00.252511 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:37:02.234779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:02.234744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:02.431920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:02.431885 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:04.582917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:04.582885 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:07.259839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:07.259806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:09.394383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:09.394347 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:37:10.264916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.264883 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c"] Nov 01 12:37:10.269136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.269112 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.318631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.318570 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"openshift-service-ca.crt\"" Nov 01 12:37:10.318822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.318766 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.318905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.318883 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.318978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.318930 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mbx\" (UniqueName: \"kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.318978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.318970 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.319084 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.319026 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.319136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.319120 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.323522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.323481 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"default-dockercfg-wsxnm\"" Nov 01 12:37:10.323674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.323631 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"kube-root-ca.crt\"" Nov 01 12:37:10.345124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.345099 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:10.364716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.364694 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"simulator-w6-f7427d-kserve-self-signed-certs\"" Nov 01 12:37:10.370469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.370447 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c"] Nov 01 12:37:10.422579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.422546 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.423135 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.423109 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mbx\" (UniqueName: \"kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.423346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.423324 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.423562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.423543 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.423732 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.423715 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.423887 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.423871 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.424101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.424072 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.424710 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.424685 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.425043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.425019 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.442550 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.442362 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.464286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.462187 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.523074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.523000 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"openshift-service-ca.crt\"" Nov 01 12:37:10.528455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.528429 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mbx\" (UniqueName: \"kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx\") pod \"simulator-w6-f7427d-kserve-5c7b84f587-l7x7c\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.581483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.581445 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:10.961110 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:10.961078 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c"] Nov 01 12:37:10.962878 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:37:10.962842 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83b07113_7770_4b13_ad2b_3c81057f4523.slice/crio-e509e2e5cafe88187207cd9608c9826681d8361aa5f12d61834ddfbcc2270590 WatchSource:0}: Error finding container e509e2e5cafe88187207cd9608c9826681d8361aa5f12d61834ddfbcc2270590: Status 404 returned error can't find the container with id e509e2e5cafe88187207cd9608c9826681d8361aa5f12d61834ddfbcc2270590 Nov 01 12:37:11.291881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:11.291848 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" event={"ID":"83b07113-7770-4b13-ad2b-3c81057f4523","Type":"ContainerStarted","Data":"594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b"} Nov 01 12:37:11.292057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:11.291890 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" event={"ID":"83b07113-7770-4b13-ad2b-3c81057f4523","Type":"ContainerStarted","Data":"e509e2e5cafe88187207cd9608c9826681d8361aa5f12d61834ddfbcc2270590"} Nov 01 12:37:11.397122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:11.397062 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podStartSLOduration=1.397045436 podStartE2EDuration="1.397045436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:37:11.396175015 +0000 UTC m=+901.146117992" watchObservedRunningTime="2026-11-01 12:37:11.397045436 +0000 UTC m=+901.146988419" Nov 01 12:37:11.582183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:11.582094 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:12.173581 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:12.173550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:17.066188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:17.066161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:20.395895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:20.395858 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:20.582110 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:20.582074 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:20.594720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:20.594694 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:21.354123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:21.354096 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:37:22.101130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:22.101090 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:26.943510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:26.943477 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:29.091259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:29.091224 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:37:30.252021 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:37:30.251989 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:37:31.853884 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:31.853849 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:35.910893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:35.910853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:36.743223 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:36.743195 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:41.717446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:41.717417 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:46.653778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:46.653740 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:51.531662 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:51.531633 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:37:56.452067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:37:56.451994 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:00.251013 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:38:00.250972 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:38:01.347463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:01.347433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:01.450792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:01.449920 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:38:04.926465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:04.926431 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:06.168467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.168436 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn"] Nov 01 12:38:06.172210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.172190 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.240661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.240628 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:06.240842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.240700 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"premium-w3-ecd640-kserve-self-signed-certs\"" Nov 01 12:38:06.240909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.240885 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn"] Nov 01 12:38:06.289530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.289499 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.289697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.289557 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.289697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.289658 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.289790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.289701 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.289830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.289786 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.289879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.289833 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbf8\" (UniqueName: \"kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.356820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.356782 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:06.391247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391215 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391266 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbf8\" (UniqueName: \"kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391356 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391392 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391439 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391482 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391787 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.391979 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.391929 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.393591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.393570 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.393769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.393754 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.444059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.443997 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbf8\" (UniqueName: \"kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8\") pod \"premium-w3-ecd640-kserve-7557b7b7d9-sptrn\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.484050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.484026 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:06.883456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:06.883426 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn"] Nov 01 12:38:06.885765 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:38:06.885727 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3419f076_07d5_4793_812b_b95a40449811.slice/crio-bfaa0732a6cb9d12b54aa13692371794474ab052384643bd51baefd3836af829 WatchSource:0}: Error finding container bfaa0732a6cb9d12b54aa13692371794474ab052384643bd51baefd3836af829: Status 404 returned error can't find the container with id bfaa0732a6cb9d12b54aa13692371794474ab052384643bd51baefd3836af829 Nov 01 12:38:07.623613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:07.623573 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" event={"ID":"3419f076-07d5-4793-812b-b95a40449811","Type":"ContainerStarted","Data":"40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222"} Nov 01 12:38:07.623613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:07.623619 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" event={"ID":"3419f076-07d5-4793-812b-b95a40449811","Type":"ContainerStarted","Data":"bfaa0732a6cb9d12b54aa13692371794474ab052384643bd51baefd3836af829"} Nov 01 12:38:07.731926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:07.731841 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podStartSLOduration=1.7318237029999999 podStartE2EDuration="1.731823703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:07.728330089 +0000 UTC m=+957.478273063" watchObservedRunningTime="2026-11-01 12:38:07.731823703 +0000 UTC m=+957.481766684" Nov 01 12:38:08.484142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:08.484113 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:11.054435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:11.054400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:14.404523 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:14.404481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:15.962128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:15.962099 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:16.485091 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:16.485062 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:16.497441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:16.497418 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:16.680230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:16.680201 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:38:20.865383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:20.865351 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:25.748500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:25.748465 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:26.876277 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:26.876241 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:30.253167 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:38:30.253122 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:38:30.659048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:30.658945 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:35.692118 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:35.692084 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:36.681607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:36.681574 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:38:40.743367 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:40.743333 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:45.729562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:45.729514 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:50.636051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:50.636014 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:52.510108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:52.510074 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:38:55.575716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:38:55.575685 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:00.267429 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:39:00.267355 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:39:00.525528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:00.525436 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:05.271498 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:05.271460 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:05.636353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:05.636255 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:10.632984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:10.632955 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:14.200528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.200490 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6"] Nov 01 12:39:14.204604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.204580 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.216452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.216428 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:39:14.274132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274103 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274137 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274254 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274320 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274391 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274444 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274431 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274533 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274603 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274561 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjkr\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.274671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.274654 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376156 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376189 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjkr\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376338 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376423 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376446 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376498 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376534 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376560 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376594 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376627 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.376881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.376850 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.377027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.377002 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.377109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.377029 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.377263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.377234 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.378685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.378659 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.378822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.378806 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.390156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.390117 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6"] Nov 01 12:39:14.429115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.429091 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.429351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.429329 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjkr\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr\") pod \"maas-default-gateway-openshift-default-7df854bf67-7gzn6\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:14.519114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:14.519085 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:15.049141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:15.049071 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6"] Nov 01 12:39:15.059235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:15.059130 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:39:15.059235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:15.059223 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:39:15.059428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:15.059269 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:39:15.099501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:15.098438 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" event={"ID":"0cdc3920-884a-4987-a830-bc24964bc85f","Type":"ContainerStarted","Data":"03f46e4d4ced9377441a10b14036dc13afa588b61f14dc302e53305af1e7d23a"} Nov 01 12:39:16.133925 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:16.133876 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" event={"ID":"0cdc3920-884a-4987-a830-bc24964bc85f","Type":"ContainerStarted","Data":"8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41"} Nov 01 12:39:16.257247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:16.257212 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:16.382321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:16.377971 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podStartSLOduration=2.377952936 podStartE2EDuration="2.377952936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:16.303642973 +0000 UTC m=+1026.053585952" watchObservedRunningTime="2026-11-01 12:39:16.377952936 +0000 UTC m=+1026.127895914" Nov 01 12:39:16.519829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:16.519793 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:16.521090 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:16.521066 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:16.521266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:16.521243 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:17.519965 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:17.519929 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:17.520468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:17.519987 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:18.520062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:18.520026 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:18.520465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:18.520084 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:19.520080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:19.520049 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:19.520478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:19.520099 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:20.520100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:20.520065 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:20.520524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:20.520143 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:21.520132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:21.520092 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:21.520552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:21.520154 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:21.853767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:21.853657 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:22.519746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:22.519709 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-7gzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 12:39:22.519918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:22.519775 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 12:39:24.248576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:24.248340 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:24.249014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:24.248708 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:24.249556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:24.249529 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:39:27.472889 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:27.472828 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:29.182383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.182347 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj"] Nov 01 12:39:29.186183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.186163 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.235116 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.235087 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-94c94b-openshift-default-dockercfg-r5knk\"" Nov 01 12:39:29.266252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266224 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266259 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266296 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266411 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266454 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266473 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4k8k\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266578 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266614 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.266690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.266678 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.284186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.284155 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj"] Nov 01 12:39:29.367626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367598 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367691 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367714 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4k8k\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367736 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367757 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367789 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367938 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.367978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.367967 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.368122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.368032 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.368504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.368211 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.368504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.368315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.368504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.368338 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.368504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.368419 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.368955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.368935 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.370110 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.370080 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.370216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.370155 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.420531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.420504 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.420657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.420576 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4k8k\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.500319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.500283 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:29.918931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.918899 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj"] Nov 01 12:39:29.920245 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:39:29.920162 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a915fc_c906_43b1_8b47_cf0b54d97404.slice/crio-e0769fad77ba73aba269e21d1a1240b57acf9be91bb7ec2664829bcc043d7cda WatchSource:0}: Error finding container e0769fad77ba73aba269e21d1a1240b57acf9be91bb7ec2664829bcc043d7cda: Status 404 returned error can't find the container with id e0769fad77ba73aba269e21d1a1240b57acf9be91bb7ec2664829bcc043d7cda Nov 01 12:39:29.922619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.922588 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:39:29.922692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.922669 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:39:29.922727 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:29.922705 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:39:30.241092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.241053 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" event={"ID":"a0a915fc-c906-43b1-8b47-cf0b54d97404","Type":"ContainerStarted","Data":"7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942"} Nov 01 12:39:30.241580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.241158 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" event={"ID":"a0a915fc-c906-43b1-8b47-cf0b54d97404","Type":"ContainerStarted","Data":"e0769fad77ba73aba269e21d1a1240b57acf9be91bb7ec2664829bcc043d7cda"} Nov 01 12:39:30.254367 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:39:30.254334 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:39:30.349842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.349772 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podStartSLOduration=1.3497555829999999 podStartE2EDuration="1.349755583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:30.348392667 +0000 UTC m=+1040.098335642" watchObservedRunningTime="2026-11-01 12:39:30.349755583 +0000 UTC m=+1040.099698558" Nov 01 12:39:30.501069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.500987 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:30.502417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.502393 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 12:39:30.502534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.502441 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 12:39:30.668142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:30.666816 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:31.501528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:31.501494 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 12:39:31.502002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:31.501552 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 12:39:32.274243 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:32.274205 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:32.501496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:32.501458 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 12:39:32.501664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:32.501511 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 12:39:33.501020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:33.500971 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 12:39:33.501189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:33.501069 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 12:39:34.500817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:34.500784 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 12:39:34.501197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:34.500852 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 12:39:35.501108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:35.501074 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 12:39:35.501534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:35.501138 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 12:39:36.567471 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:36.567435 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:36.567990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:36.567738 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:36.568641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:36.568621 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:39:37.157832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:37.157796 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:42.347468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:42.347438 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:43.872090 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:43.872059 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:47.249174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:47.249135 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:52.249086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:52.249054 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:57.067475 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:57.067438 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:39:59.513274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.513244 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97"] Nov 01 12:39:59.517024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.517001 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.615207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.615168 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97"] Nov 01 12:39:59.687025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.686983 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687123 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687144 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687165 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687195 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687263 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687360 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687390 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.687631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.687478 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnchs\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788499 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788583 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788599 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788617 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788635 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788675 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788827 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788879 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.788945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.788935 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnchs\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.789086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.789022 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.789145 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.789099 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.789145 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.789116 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.789254 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.789170 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.789381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.789363 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.791146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.791124 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.791373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.791351 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.840946 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.840922 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:39:59.841048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:39:59.840983 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnchs\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:40:00.130567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:00.130492 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:40:00.196986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:00.196956 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:40:00.255052 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:40:00.255017 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:40:00.613331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:00.611101 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97"] Nov 01 12:40:00.631019 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:40:00.630988 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28af2f2e_7971_45d7_aa4d_bf8e5afc89b4.slice/crio-f6d49241b8bafa72f9d72bd2c6976d3c5055c77a53829c6d78aed5d672cfa0e4 WatchSource:0}: Error finding container f6d49241b8bafa72f9d72bd2c6976d3c5055c77a53829c6d78aed5d672cfa0e4: Status 404 returned error can't find the container with id f6d49241b8bafa72f9d72bd2c6976d3c5055c77a53829c6d78aed5d672cfa0e4 Nov 01 12:40:00.635276 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:00.635230 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:40:00.635500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:00.635476 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:40:00.635661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:00.635641 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:40:01.479220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:01.479178 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" event={"ID":"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4","Type":"ContainerStarted","Data":"d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b"} Nov 01 12:40:01.479520 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:01.479494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" event={"ID":"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4","Type":"ContainerStarted","Data":"f6d49241b8bafa72f9d72bd2c6976d3c5055c77a53829c6d78aed5d672cfa0e4"} Nov 01 12:40:01.705348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:01.702223 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podStartSLOduration=2.702199944 podStartE2EDuration="2.702199944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:01.650037143 +0000 UTC m=+1071.399980124" watchObservedRunningTime="2026-11-01 12:40:01.702199944 +0000 UTC m=+1071.452142923" Nov 01 12:40:02.131779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:02.131740 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:40:02.133917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:02.133878 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 12:40:02.134181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:02.134133 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 12:40:02.343830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:02.343795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:03.130935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:03.130900 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 12:40:03.131370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:03.130956 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 12:40:04.131856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:04.131817 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 12:40:04.132258 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:04.131907 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 12:40:05.130975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:05.130936 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 12:40:05.131184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:05.131002 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 12:40:05.631057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:05.631025 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:06.131516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:06.131485 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 12:40:06.131699 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:06.131536 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 12:40:07.131585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:07.131537 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 12:40:07.131991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:07.131614 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 12:40:07.574582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:07.574550 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:08.684610 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:08.684533 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:40:08.685217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:08.685195 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:40:08.686100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:08.686075 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:40:10.120318 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:10.120269 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:10.230568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:10.230539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:12.398374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:12.398339 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:14.624310 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.624270 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq"] Nov 01 12:40:14.628717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.628695 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.676851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.676822 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5b1bd4-openshift-default-dockercfg-4g7mc\"" Nov 01 12:40:14.726624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.726596 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq"] Nov 01 12:40:14.734293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734265 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734335 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734429 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734474 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734624 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734675 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734785 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734770 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734877 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734862 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.734933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.734917 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4flp\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836005 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4flp\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836077 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836104 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836136 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836167 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836233 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836274 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836350 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836446 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836571 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836795 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.836888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.836826 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.837111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.837086 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.837208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.837111 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.838516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.838496 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.838828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.838810 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.889451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.889394 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.889655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.889637 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4flp\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:14.941720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:14.941696 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:15.359053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.359027 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq"] Nov 01 12:40:15.360889 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:40:15.360861 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c195962_b11f_4f79_ae40_96ca0f1bd60e.slice/crio-c06cf21bbae7fdb4c027e2d3538acdf83cc0ce3e89a1da9436293a504551a976 WatchSource:0}: Error finding container c06cf21bbae7fdb4c027e2d3538acdf83cc0ce3e89a1da9436293a504551a976: Status 404 returned error can't find the container with id c06cf21bbae7fdb4c027e2d3538acdf83cc0ce3e89a1da9436293a504551a976 Nov 01 12:40:15.363363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.363328 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:40:15.363434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.363393 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:40:15.363434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.363420 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:40:15.637668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.637576 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" event={"ID":"8c195962-b11f-4f79-ae40-96ca0f1bd60e","Type":"ContainerStarted","Data":"d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487"} Nov 01 12:40:15.637668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.637641 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" event={"ID":"8c195962-b11f-4f79-ae40-96ca0f1bd60e","Type":"ContainerStarted","Data":"c06cf21bbae7fdb4c027e2d3538acdf83cc0ce3e89a1da9436293a504551a976"} Nov 01 12:40:15.746867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.746807 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podStartSLOduration=1.7467911379999999 podStartE2EDuration="1.746791138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:15.744177778 +0000 UTC m=+1085.494120756" watchObservedRunningTime="2026-11-01 12:40:15.746791138 +0000 UTC m=+1085.496734320" Nov 01 12:40:15.953522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:15.953413 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:16.961322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:16.957700 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 12:40:16.961322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:16.957762 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 12:40:17.272189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:17.272152 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:17.964324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:17.963626 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 12:40:17.964324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:17.963687 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 12:40:18.961325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:18.951648 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 12:40:18.961325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:18.951716 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 12:40:19.960321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:19.950616 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 12:40:19.960321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:19.950680 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 12:40:20.955326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:20.945606 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 12:40:20.955326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:20.945695 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 12:40:22.115313 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:22.115264 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:22.115823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:22.115797 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:22.116931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:22.116905 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 12:40:22.244019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:22.243992 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:27.146342 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:27.146288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:30.255041 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:40:30.255005 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:40:32.061948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:32.061917 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:32.324030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:32.323947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:33.035975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:33.035945 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:40:37.069959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:37.069919 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:38.424184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.424155 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62"] Nov 01 12:40:38.428176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.428158 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.481965 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.481939 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62"] Nov 01 12:40:38.498267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.498224 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.498413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.498355 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.498413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.498378 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.498529 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.498418 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.498618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.498594 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkng\" (UniqueName: \"kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.498678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.498655 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.508089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.508052 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:38.524764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.524748 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"distinct-w6-f7427d-kserve-self-signed-certs\"" Nov 01 12:40:38.600948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.600909 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.601244 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.601225 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.601487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.601468 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.601607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.601593 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.601753 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.601737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.601934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.601917 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkng\" (UniqueName: \"kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.602916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.602892 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.603325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.603289 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.603839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.603818 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.634697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.634657 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.643002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.636221 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.655336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.655287 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkng\" (UniqueName: \"kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng\") pod \"distinct-w6-f7427d-kserve-85d5b8778f-wxm62\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:38.748804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:38.748778 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:39.195871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:39.195850 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62"] Nov 01 12:40:39.198631 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:40:39.198603 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc60a3282_fa6e_4935_a053_196324513375.slice/crio-218b424a954acf736e5565a050bc6e69568a3dc2a5816d1ac21082c7a3e3bc26 WatchSource:0}: Error finding container 218b424a954acf736e5565a050bc6e69568a3dc2a5816d1ac21082c7a3e3bc26: Status 404 returned error can't find the container with id 218b424a954acf736e5565a050bc6e69568a3dc2a5816d1ac21082c7a3e3bc26 Nov 01 12:40:39.846515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:39.846483 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" event={"ID":"c60a3282-fa6e-4935-a053-196324513375","Type":"ContainerStarted","Data":"edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad"} Nov 01 12:40:39.846886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:39.846525 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" event={"ID":"c60a3282-fa6e-4935-a053-196324513375","Type":"ContainerStarted","Data":"218b424a954acf736e5565a050bc6e69568a3dc2a5816d1ac21082c7a3e3bc26"} Nov 01 12:40:39.951851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:39.951809 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podStartSLOduration=1.95179189 podStartE2EDuration="1.95179189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:39.950812085 +0000 UTC m=+1109.700755062" watchObservedRunningTime="2026-11-01 12:40:39.95179189 +0000 UTC m=+1109.701734867" Nov 01 12:40:40.749724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:40.749696 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:41.576354 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:41.576288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:41.987552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:41.987505 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:46.842553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:46.842520 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:48.749900 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:48.749867 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:48.762500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:48.762476 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:48.917197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:48.917175 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:40:50.044748 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.044715 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp"] Nov 01 12:40:50.048899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.048879 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.092873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.092845 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsvp\" (UniqueName: \"kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.093022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.092880 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.093022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.093010 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.093113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.093039 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.093113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.093081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.093249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.093222 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.117390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.117365 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:50.123415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.123392 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp"] Nov 01 12:40:50.144357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.144293 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"distinct-w0-94c94b-kserve-self-signed-certs\"" Nov 01 12:40:50.223002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.222964 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.238645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.238601 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsvp\" (UniqueName: \"kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.238907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.238885 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.239348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.239272 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.239522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.239505 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.239688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.239673 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.240747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.240725 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.241154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.241134 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.241840 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.241820 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.246224 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.246203 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.249915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.249894 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.294470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.294440 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsvp\" (UniqueName: \"kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp\") pod \"distinct-w0-94c94b-kserve-544947d9fd-tczrp\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.361996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.361927 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:50.811493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.811466 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp"] Nov 01 12:40:50.813143 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:40:50.813111 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7f1ee28_a99a_4dea_9890_4474f8c0e312.slice/crio-0267017e2fe90ba20493a5682b71fee8def00a60c06dd268dbacfb029bd44898 WatchSource:0}: Error finding container 0267017e2fe90ba20493a5682b71fee8def00a60c06dd268dbacfb029bd44898: Status 404 returned error can't find the container with id 0267017e2fe90ba20493a5682b71fee8def00a60c06dd268dbacfb029bd44898 Nov 01 12:40:50.918402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.918368 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" event={"ID":"e7f1ee28-a99a-4dea-9890-4474f8c0e312","Type":"ContainerStarted","Data":"2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b"} Nov 01 12:40:50.918402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:50.918414 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" event={"ID":"e7f1ee28-a99a-4dea-9890-4474f8c0e312","Type":"ContainerStarted","Data":"0267017e2fe90ba20493a5682b71fee8def00a60c06dd268dbacfb029bd44898"} Nov 01 12:40:51.025884 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:51.025836 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podStartSLOduration=1.025821838 podStartE2EDuration="1.025821838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:51.02468876 +0000 UTC m=+1120.774631738" watchObservedRunningTime="2026-11-01 12:40:51.025821838 +0000 UTC m=+1120.775764815" Nov 01 12:40:51.362386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:51.362356 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:40:51.755868 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:51.755819 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:56.654968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:56.654899 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:58.115369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.114235 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:40:58.276435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.276397 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99"] Nov 01 12:40:58.282630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.282602 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.322646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.322614 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99"] Nov 01 12:40:58.334516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.334481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:40:58.341571 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.341541 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.341690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.341582 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.341772 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.341748 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.341866 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.341848 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnsj\" (UniqueName: \"kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.342010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.341990 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.342115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.342014 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.376047 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.375965 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"distinct-w3-ecd640-kserve-self-signed-certs\"" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.492426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.492839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.492950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnsj\" (UniqueName: \"kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.493180 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.493212 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.493318 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.504165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.493345 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.506714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.506036 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.506714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.506374 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.565390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.565264 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.567476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.567446 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.596064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.596000 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnsj\" (UniqueName: \"kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj\") pod \"distinct-w3-ecd640-kserve-5d999d4c76-wqc99\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:58.598586 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:58.598551 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:40:59.149538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:40:59.149502 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99"] Nov 01 12:41:00.039099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:00.038113 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" event={"ID":"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3","Type":"ContainerStarted","Data":"136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18"} Nov 01 12:41:00.039099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:00.038171 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" event={"ID":"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3","Type":"ContainerStarted","Data":"e4b1a33b14fb8ac84b1a7ff812c2d6920e41a7173835a730378b7146c816372e"} Nov 01 12:41:00.591323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:00.585205 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:41:00.608321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:00.606370 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:41:00.714335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:00.713068 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:41:00.721663 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:41:00.721629 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:41:00.824342 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:00.824256 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podStartSLOduration=2.824237274 podStartE2EDuration="2.824237274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:00.171166645 +0000 UTC m=+1129.921109625" watchObservedRunningTime="2026-11-01 12:41:00.824237274 +0000 UTC m=+1130.574180251" Nov 01 12:41:01.057882 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:01.057857 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:41:02.064803 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:02.064771 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:05.965404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:05.965366 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:07.198823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:07.198786 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:08.601054 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:08.600973 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:41:08.613479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:08.613458 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:41:09.101414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:09.101375 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:41:10.994583 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:10.994545 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:12.160968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:12.160935 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:15.074975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.074937 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v"] Nov 01 12:41:15.080605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.080583 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.141438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.141410 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t"] Nov 01 12:41:15.145572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.145545 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.157814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.157787 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.157999 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158045 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158082 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158159 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158196 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158239 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv7c\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158269 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158349 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s929h\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158452 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158473 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158528 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158582 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158621 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158686 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158721 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.158909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158738 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.158909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.158754 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.185219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.185193 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v"] Nov 01 12:41:15.218549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.218523 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t"] Nov 01 12:41:15.260364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260331 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv7c\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.260506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260368 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.260506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260393 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s929h\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.260506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260422 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.260506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260444 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.260506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260482 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.260764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260550 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.260764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260607 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.260764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260658 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.260764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260701 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.260764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260736 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260769 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260798 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260821 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260854 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260855 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260907 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.260958 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261002 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261028 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261062 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261258 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261423 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261515 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261532 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.261742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261564 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261910 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261788 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.261972 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.261918 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.263274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.263254 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.263714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.263680 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.263798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.263690 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.263798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.263765 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.314043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.314016 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv7c\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.314182 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.314088 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s929h\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.314255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.314226 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.314369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.314348 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.393943 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.393889 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:15.460496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.460419 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:15.811695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.811662 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v"] Nov 01 12:41:15.850619 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:41:15.850594 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fed01f_aa7e_4698_9b9a_29a8e8f94f89.slice/crio-286b33c90f8136ac3034a375bb5c5c7c16b413fb2a90aa7803521ec312d4ed6b WatchSource:0}: Error finding container 286b33c90f8136ac3034a375bb5c5c7c16b413fb2a90aa7803521ec312d4ed6b: Status 404 returned error can't find the container with id 286b33c90f8136ac3034a375bb5c5c7c16b413fb2a90aa7803521ec312d4ed6b Nov 01 12:41:15.852890 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.852859 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:41:15.852965 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.852933 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:41:15.853005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.852964 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:41:15.871532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.871489 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:41:15.871973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.871941 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:41:15.872075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.872015 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:41:15.873698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:15.873604 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t"] Nov 01 12:41:16.148624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.148534 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" event={"ID":"311404f1-fbc2-4e49-94c0-4128cea90c0c","Type":"ContainerStarted","Data":"9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1"} Nov 01 12:41:16.148624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.148595 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" event={"ID":"311404f1-fbc2-4e49-94c0-4128cea90c0c","Type":"ContainerStarted","Data":"0722f65f460cee20e8b77db01aa9ed49554f518aa3df5a6599d4f8f9a7407787"} Nov 01 12:41:16.150465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.150436 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" event={"ID":"83fed01f-aa7e-4698-9b9a-29a8e8f94f89","Type":"ContainerStarted","Data":"e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5"} Nov 01 12:41:16.150586 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.150486 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" event={"ID":"83fed01f-aa7e-4698-9b9a-29a8e8f94f89","Type":"ContainerStarted","Data":"286b33c90f8136ac3034a375bb5c5c7c16b413fb2a90aa7803521ec312d4ed6b"} Nov 01 12:41:16.260080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.260017 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podStartSLOduration=1.260002493 podStartE2EDuration="1.260002493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:16.256875189 +0000 UTC m=+1146.006818189" watchObservedRunningTime="2026-11-01 12:41:16.260002493 +0000 UTC m=+1146.009945471" Nov 01 12:41:16.374800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.371041 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podStartSLOduration=1.371021172 podStartE2EDuration="1.371021172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:16.36985259 +0000 UTC m=+1146.119795569" watchObservedRunningTime="2026-11-01 12:41:16.371021172 +0000 UTC m=+1146.120964151" Nov 01 12:41:16.400927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.395153 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:16.404322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.401425 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:16.404322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.401489 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:16.463514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.463471 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:16.463935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.463896 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:16.464186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:16.464124 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:17.095321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:17.092856 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:17.394954 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:17.394836 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:17.394954 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:17.394918 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:17.464678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:17.464641 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:17.464930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:17.464902 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:18.394967 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:18.394931 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:18.395568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:18.395538 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:18.470324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:18.462633 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:18.470324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:18.462686 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:19.394759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:19.394729 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:19.395004 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:19.394983 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:19.470324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:19.469692 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:19.470324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:19.469750 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:20.394893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:20.394857 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:20.395080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:20.394916 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:20.463326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:20.461692 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:20.463326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:20.461750 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:21.395122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:21.394369 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:21.395122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:21.394440 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:21.462712 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:21.462674 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:21.462989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:21.462960 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:22.333769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:22.333732 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:22.404936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:22.403031 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 12:41:22.404936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:22.403096 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 12:41:22.467323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:22.466452 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 12:41:22.467323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:22.466545 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 12:41:23.399127 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:23.399091 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:23.399569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:23.399547 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:23.400388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:23.400368 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:41:23.465230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:23.465206 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:23.465631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:23.465482 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:23.466196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:23.466180 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:41:26.778210 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:26.778176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:26.931819 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:26.931792 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:27.258010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:27.257978 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:30.252208 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:41:30.252173 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:41:32.139739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:32.139706 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:34.774116 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.774079 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf"] Nov 01 12:41:34.778459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.778434 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.845091 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.845059 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:34.849124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.849100 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf"] Nov 01 12:41:34.873778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.873750 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"openshift-service-ca.crt\"" Nov 01 12:41:34.873908 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.873868 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"kube-root-ca.crt\"" Nov 01 12:41:34.873978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.873956 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"distinct-2-w2-c7206f-kserve-self-signed-certs\"" Nov 01 12:41:34.873978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.873957 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"default-dockercfg-8rwpt\"" Nov 01 12:41:34.892770 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.892748 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.892870 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.892814 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.892911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.892891 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.893015 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.892996 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.893063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.893047 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.893131 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.893118 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92q5s\" (UniqueName: \"kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994100 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994254 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994188 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994278 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994396 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994466 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994445 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994496 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92q5s\" (UniqueName: \"kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994803 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994971 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994836 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.994971 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.994875 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.996626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.996604 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:34.996925 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:34.996906 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:35.046854 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:35.046795 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92q5s\" (UniqueName: \"kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s\") pod \"distinct-2-w2-c7206f-kserve-84f878b8-7gfqf\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:35.090336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:35.090312 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:35.506878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:35.506855 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf"] Nov 01 12:41:35.508201 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:41:35.508177 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da1edfe_7017_482e_bb96_5aaf5d68bdd5.slice/crio-cc7e950d47a9490ade1e7378329269ec05cd74461de8af9bcee711b24b5380c5 WatchSource:0}: Error finding container cc7e950d47a9490ade1e7378329269ec05cd74461de8af9bcee711b24b5380c5: Status 404 returned error can't find the container with id cc7e950d47a9490ade1e7378329269ec05cd74461de8af9bcee711b24b5380c5 Nov 01 12:41:36.317980 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:36.317947 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" event={"ID":"5da1edfe-7017-482e-bb96-5aaf5d68bdd5","Type":"ContainerStarted","Data":"ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651"} Nov 01 12:41:36.317980 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:36.317986 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" event={"ID":"5da1edfe-7017-482e-bb96-5aaf5d68bdd5","Type":"ContainerStarted","Data":"cc7e950d47a9490ade1e7378329269ec05cd74461de8af9bcee711b24b5380c5"} Nov 01 12:41:36.428960 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:36.428897 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podStartSLOduration=2.428878589 podStartE2EDuration="2.428878589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:36.425926473 +0000 UTC m=+1166.175869450" watchObservedRunningTime="2026-11-01 12:41:36.428878589 +0000 UTC m=+1166.178821568" Nov 01 12:41:37.056040 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:37.056008 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:37.090782 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:37.090754 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:39.865049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:39.865016 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:41:41.938499 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:41.938437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:45.090577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:45.090546 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:45.103394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:45.103370 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:45.389137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:45.389067 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:41:46.870569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:46.870528 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:52.146924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:52.146884 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:41:57.142715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:41:57.142675 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:00.252712 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:42:00.252674 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:42:02.043349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:02.043275 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:06.296915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:06.296878 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:07.151280 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:07.151239 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:10.733558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:10.733527 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:42:11.841479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:11.841448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:12.373830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:12.373792 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:12.487695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:12.487664 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:15.370423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.370386 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm"] Nov 01 12:42:15.374920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.374897 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.473048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.473019 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm"] Nov 01 12:42:15.508799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.508772 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.508930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.508829 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509066 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509043 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509097 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509144 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx559\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509201 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509175 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509236 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509200 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509236 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509215 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.509389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.509369 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.610834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.610802 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.610975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.610844 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.610975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.610875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx559\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.610975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.610903 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.610975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.610928 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.610975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.610950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611024 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611085 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611153 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611226 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611287 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611340 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611589 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.611678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.611658 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.614284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.614262 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.614692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.614673 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.663607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.663561 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.663790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.663770 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx559\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:15.688016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:15.687991 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:16.107957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.107931 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm"] Nov 01 12:42:16.109360 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:42:16.109296 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78a0a40_774d_442a_ae7b_11e317071bc3.slice/crio-27b33243a5ea23684315e831dc9682efe90e27a253ee77b292f20239b98d7248 WatchSource:0}: Error finding container 27b33243a5ea23684315e831dc9682efe90e27a253ee77b292f20239b98d7248: Status 404 returned error can't find the container with id 27b33243a5ea23684315e831dc9682efe90e27a253ee77b292f20239b98d7248 Nov 01 12:42:16.111655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.111620 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:42:16.111746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.111688 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:42:16.111746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.111716 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:42:16.555530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.555491 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" event={"ID":"d78a0a40-774d-442a-ae7b-11e317071bc3","Type":"ContainerStarted","Data":"eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8"} Nov 01 12:42:16.555930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.555554 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" event={"ID":"d78a0a40-774d-442a-ae7b-11e317071bc3","Type":"ContainerStarted","Data":"27b33243a5ea23684315e831dc9682efe90e27a253ee77b292f20239b98d7248"} Nov 01 12:42:16.688841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.688808 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:16.710020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.709972 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podStartSLOduration=1.709959134 podStartE2EDuration="1.709959134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:16.662832277 +0000 UTC m=+1206.412775259" watchObservedRunningTime="2026-11-01 12:42:16.709959134 +0000 UTC m=+1206.459902111" Nov 01 12:42:16.728831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:16.728776 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:17.690539 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:17.690507 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 12:42:17.690975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:17.690562 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 12:42:18.688535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:18.688501 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 12:42:18.688730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:18.688553 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 12:42:19.688751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:19.688714 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 12:42:19.689154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:19.688769 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 12:42:20.689178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:20.689139 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 12:42:20.689580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:20.689198 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 12:42:21.573197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:21.573160 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:21.693683 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:21.693635 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 12:42:21.694363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:21.694292 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 12:42:22.688629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:22.688592 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 12:42:22.688810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:22.688668 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 12:42:23.694037 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:23.693962 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:23.694434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:23.694420 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:23.695142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:23.695124 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:42:26.684851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:26.684806 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:30.251196 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:42:30.251161 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:42:31.517867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:31.517838 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:36.349573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:36.349537 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:41.237027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:41.236990 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:42.801364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:42.801332 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:42:45.717714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.717677 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6"] Nov 01 12:42:45.722088 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.722066 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.819697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.819665 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6"] Nov 01 12:42:45.842065 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842021 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9sdp\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842090 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842243 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842275 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842334 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842403 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842455 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842513 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.842643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.842620 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943293 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943393 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943417 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943451 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9sdp\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943479 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943588 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943604 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943638 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.943769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.943686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.944432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.944403 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.944768 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.944743 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.945103 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.945080 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.945230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.945177 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.945469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.945450 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.946589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.946565 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.946809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.946788 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.997477 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.997443 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9sdp\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:45.997614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:45.997506 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:46.039499 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.039479 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:46.154336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.154291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:46.454428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.454406 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6"] Nov 01 12:42:46.459194 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:42:46.459168 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f659fa_cd16_4857_a348_3ed5d08c2714.slice/crio-3de601aa14d0d59ece7437de0a70003d02e4e3ff2a10a77933cfc47955d4c5bb WatchSource:0}: Error finding container 3de601aa14d0d59ece7437de0a70003d02e4e3ff2a10a77933cfc47955d4c5bb: Status 404 returned error can't find the container with id 3de601aa14d0d59ece7437de0a70003d02e4e3ff2a10a77933cfc47955d4c5bb Nov 01 12:42:46.462083 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.462050 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:42:46.462164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.462114 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:42:46.462164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.462140 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:42:46.752853 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.752821 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" event={"ID":"76f659fa-cd16-4857-a348-3ed5d08c2714","Type":"ContainerStarted","Data":"8eaeb03574ab76cefe8898c9851db0d62bfebdc7694c3c1cfebb6b506d5e5c93"} Nov 01 12:42:46.753234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.752878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" event={"ID":"76f659fa-cd16-4857-a348-3ed5d08c2714","Type":"ContainerStarted","Data":"3de601aa14d0d59ece7437de0a70003d02e4e3ff2a10a77933cfc47955d4c5bb"} Nov 01 12:42:46.829215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.829178 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm"] Nov 01 12:42:46.833539 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.833519 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.870057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.870003 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podStartSLOduration=1.869985652 podStartE2EDuration="1.869985652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:46.864123615 +0000 UTC m=+1236.614066593" watchObservedRunningTime="2026-11-01 12:42:46.869985652 +0000 UTC m=+1236.619928630" Nov 01 12:42:46.889074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889048 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"distinct-2-w6-f7427d-kserve-self-signed-certs\"" Nov 01 12:42:46.889397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889369 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm"] Nov 01 12:42:46.889560 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889541 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.889626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889599 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.889681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889667 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kzz\" (UniqueName: \"kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.889802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889788 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.889865 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889826 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.889865 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.889857 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.911175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.911147 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:46.991399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.991360 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.991681 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kzz\" (UniqueName: \"kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.991854 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.991917 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.991979 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.992193 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.993661 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.993932 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:46.995322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:46.994158 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:47.008223 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.008152 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:47.037724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.037635 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:47.043763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.043682 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:47.048903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.047120 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:47.048903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.047208 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:47.072805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.072768 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kzz\" (UniqueName: \"kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz\") pod \"distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:47.146459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.146432 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:47.717205 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.717183 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm"] Nov 01 12:42:47.718977 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:42:47.718949 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bdffd62_0423_421a_a682_2165a6e00409.slice/crio-3ce26426d81d7ede6b6301ef09e741165b981bf54433aebdfea04accff36b466 WatchSource:0}: Error finding container 3ce26426d81d7ede6b6301ef09e741165b981bf54433aebdfea04accff36b466: Status 404 returned error can't find the container with id 3ce26426d81d7ede6b6301ef09e741165b981bf54433aebdfea04accff36b466 Nov 01 12:42:47.772540 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:47.772507 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" event={"ID":"5bdffd62-0423-421a-a682-2165a6e00409","Type":"ContainerStarted","Data":"3ce26426d81d7ede6b6301ef09e741165b981bf54433aebdfea04accff36b466"} Nov 01 12:42:48.040502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:48.040474 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:48.040672 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:48.040529 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:48.780552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:48.780509 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" event={"ID":"5bdffd62-0423-421a-a682-2165a6e00409","Type":"ContainerStarted","Data":"b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66"} Nov 01 12:42:48.903085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:48.903039 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podStartSLOduration=2.903023821 podStartE2EDuration="2.903023821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:48.902394499 +0000 UTC m=+1238.652337478" watchObservedRunningTime="2026-11-01 12:42:48.903023821 +0000 UTC m=+1238.652966799" Nov 01 12:42:49.049378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:49.041680 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:49.049378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:49.041862 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:49.152793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:49.147597 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:50.040103 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:50.040068 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:50.040547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:50.040123 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:51.040467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:51.040430 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:51.040838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:51.040477 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:51.189810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:51.189778 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:52.040652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:52.040619 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:52.041036 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:52.040671 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:53.040416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:53.040383 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 12:42:53.040578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:53.040436 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 12:42:54.086634 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:54.086585 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:54.087478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:54.087448 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:54.088390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:54.088365 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:42:56.156721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:56.156676 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:56.272410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:56.272376 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:42:57.147358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:57.147293 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:57.160292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:57.160266 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:42:57.842834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:42:57.842806 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:43:00.251470 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:43:00.251435 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:43:00.709020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.708930 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv"] Nov 01 12:43:00.713134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.713111 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.809923 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.809874 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810044 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810075 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2kzr\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810193 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810181 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810249 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810331 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810373 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810429 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.810487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.810448 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.901436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.901405 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv"] Nov 01 12:43:00.911453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911424 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.911613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911519 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.911613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911590 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.911794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911641 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.911794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911684 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.911794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911708 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.911794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911743 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.912003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911871 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.912003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911895 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2kzr\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.912003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911902 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.912003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.911920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.912586 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.912562 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.912586 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.912573 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.913122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.913063 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.914499 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.914475 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.915189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.915168 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.964354 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.964263 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:00.964675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:00.964657 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2kzr\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:01.027252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.027223 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:01.051052 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.051028 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:01.450241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.450213 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv"] Nov 01 12:43:01.451415 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:43:01.451391 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a304bf5_a631_4fe9_b47c_6df19ef8b9d8.slice/crio-74ee3841da1596559f67820437ab29f1dd85d8d5a44dfe5bf28031a861f5f667 WatchSource:0}: Error finding container 74ee3841da1596559f67820437ab29f1dd85d8d5a44dfe5bf28031a861f5f667: Status 404 returned error can't find the container with id 74ee3841da1596559f67820437ab29f1dd85d8d5a44dfe5bf28031a861f5f667 Nov 01 12:43:01.453906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.453869 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:01.454022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.453950 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:01.454085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.454021 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:01.860013 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.859973 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" event={"ID":"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8","Type":"ContainerStarted","Data":"4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7"} Nov 01 12:43:01.860191 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:01.860044 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" event={"ID":"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8","Type":"ContainerStarted","Data":"74ee3841da1596559f67820437ab29f1dd85d8d5a44dfe5bf28031a861f5f667"} Nov 01 12:43:02.022017 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:02.021956 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podStartSLOduration=2.021939618 podStartE2EDuration="2.021939618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:02.018220194 +0000 UTC m=+1251.768163172" watchObservedRunningTime="2026-11-01 12:43:02.021939618 +0000 UTC m=+1251.771882595" Nov 01 12:43:02.028142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:02.028118 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:02.029823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:02.029801 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:02.029937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:02.029849 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:02.459769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:02.459733 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:03.027872 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:03.027839 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:03.028042 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:03.027898 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:04.027506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:04.027470 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:04.028000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:04.027531 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:05.028172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:05.028135 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:05.028575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:05.028187 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:06.027695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:06.027661 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:06.027884 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:06.027713 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:06.222603 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:06.222570 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:06.713926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:06.713893 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:07.027896 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:07.027863 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:07.028083 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:07.027939 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:07.377171 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:07.377060 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:08.028048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:08.028012 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 12:43:08.028208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:08.028078 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 12:43:09.032276 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:09.032244 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:09.032693 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:09.032524 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:09.033128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:09.033109 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:43:11.341822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:11.341790 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:15.235610 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.234932 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n"] Nov 01 12:43:15.244159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.244131 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.287950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.287917 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n"] Nov 01 12:43:15.318598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.318563 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:15.327981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.327765 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"default-dockercfg-rrgs4\"" Nov 01 12:43:15.327981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.327919 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"distinct-2-w5-9514b0-kserve-self-signed-certs\"" Nov 01 12:43:15.339753 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.339581 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"openshift-service-ca.crt\"" Nov 01 12:43:15.339753 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.339701 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"kube-root-ca.crt\"" Nov 01 12:43:15.429346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.429285 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.429741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.429698 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.430009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.429979 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.430211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.430192 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bsd\" (UniqueName: \"kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.430539 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.430519 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.430759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.430725 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.533229 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.533329 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.533408 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.533453 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bsd\" (UniqueName: \"kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.533629 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.533676 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.534870 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.535110 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.540371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.535744 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.568320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.566005 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.572321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.569686 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.600327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.596321 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bsd\" (UniqueName: \"kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd\") pod \"distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:15.883376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:15.883160 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:16.500678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:16.500477 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": context deadline exceeded" Nov 01 12:43:16.704462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:16.696899 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:43:16.704462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:16.699154 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:16.775387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:16.771424 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:16.952324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:16.951373 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:16.990637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:16.988398 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.053077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.051405 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.094235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.092252 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.179790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.179744 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n"] Nov 01 12:43:17.305949 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.239011 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.388674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.353451 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.397162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.397111 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:43:17.502104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.502059 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.557686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.557602 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:17.719379 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.712958 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.739329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.735690 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.756388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.756354 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:43:17.832384 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.784816 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.841378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.836992 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:43:17.841378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.785233 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:17.841378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.837830 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:43:17.862776 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:17.862731 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:18.010645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.004528 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:43:18.088615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.088443 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:18.090419 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.090370 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:18.104339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.092697 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:43:18.104339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.092764 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:43:18.149386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.105051 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:18.149386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.105564 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:43:18.149386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.126500 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 12:43:18.149386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.127892 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:43:18.239221 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.239165 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 12:43:18.240121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.240093 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 01 12:43:18.274778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.274751 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:43:18.458140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.458046 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:43:18.462323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.458679 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:43:18.705326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.647964 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:43:18.792321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.735172 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:18.792321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.756758 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:18.815617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.815576 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" event={"ID":"97ed692d-230c-4d02-8475-eec80bab204f","Type":"ContainerStarted","Data":"c977ce32b15f8319913c9deeeb05aa29730a2c2c64e70e8e5586d126ad5460ac"} Nov 01 12:43:18.815901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.815880 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:43:18.816091 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.816066 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:43:18.818510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.818474 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:43:18.835964 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.824077 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:43:18.835964 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.824123 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:43:18.864831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.864741 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:18.979332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:18.926725 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:43:19.300349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.300294 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:43:19.300538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.300376 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:43:19.300538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.300412 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:43:19.300538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.300447 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:43:19.328843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.328798 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:19.392791 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.392736 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:43:19.794719 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.794562 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" event={"ID":"97ed692d-230c-4d02-8475-eec80bab204f","Type":"ContainerStarted","Data":"cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221"} Nov 01 12:43:19.883662 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:19.883631 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:20.724940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:20.724890 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podStartSLOduration=5.724874367 podStartE2EDuration="5.724874367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:20.722417681 +0000 UTC m=+1270.472360659" watchObservedRunningTime="2026-11-01 12:43:20.724874367 +0000 UTC m=+1270.474817355" Nov 01 12:43:23.344294 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:23.344263 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:25.884030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:25.883999 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:25.897027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:25.897004 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:26.850245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:26.850216 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:43:28.271817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:28.271779 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:30.251830 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:43:30.251792 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:43:30.970933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:30.970895 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj"] Nov 01 12:43:30.974893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:30.974868 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.069159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.069124 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj"] Nov 01 12:43:31.093848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.093814 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.093878 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094013 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094192 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094261 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094290 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcsv\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094335 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094418 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.094451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.094436 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.195746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195713 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.195746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195750 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcsv\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195777 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195818 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195834 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195915 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.195977 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196411 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196211 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196265 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196628 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196579 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196653 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196725 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.196901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.196882 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.198339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.198317 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.198450 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.198432 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.249752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.249725 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.249885 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.249793 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcsv\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.289749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.289723 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:31.708294 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.708270 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj"] Nov 01 12:43:31.710288 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:43:31.710260 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode410b77d_0691_47ef_a10d_b4cf73d4ce56.slice/crio-21270cb2a2746d2bcce106a04f5e4939daf71b806a69239714bae7527796a132 WatchSource:0}: Error finding container 21270cb2a2746d2bcce106a04f5e4939daf71b806a69239714bae7527796a132: Status 404 returned error can't find the container with id 21270cb2a2746d2bcce106a04f5e4939daf71b806a69239714bae7527796a132 Nov 01 12:43:31.712825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.712791 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:31.712939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.712858 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:31.712939 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.712887 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:31.876187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.876147 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" event={"ID":"e410b77d-0691-47ef-a10d-b4cf73d4ce56","Type":"ContainerStarted","Data":"a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a"} Nov 01 12:43:31.876377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.876219 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" event={"ID":"e410b77d-0691-47ef-a10d-b4cf73d4ce56","Type":"ContainerStarted","Data":"21270cb2a2746d2bcce106a04f5e4939daf71b806a69239714bae7527796a132"} Nov 01 12:43:31.986104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:31.986040 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podStartSLOduration=1.986023596 podStartE2EDuration="1.986023596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:31.985056489 +0000 UTC m=+1281.734999468" watchObservedRunningTime="2026-11-01 12:43:31.986023596 +0000 UTC m=+1281.735966574" Nov 01 12:43:32.290787 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:32.290690 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:32.292576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:32.292551 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:32.292676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:32.292602 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:33.263168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:33.263136 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:33.291011 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:33.290980 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:33.291393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:33.291040 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:34.290607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:34.290575 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:34.290786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:34.290635 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:35.290659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:35.290622 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:35.291068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:35.290686 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:36.290844 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:36.290806 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:36.291235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:36.290874 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:37.290828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:37.290791 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:37.291046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:37.290861 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:38.290899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:38.290867 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 12:43:38.291063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:38.290939 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 12:43:38.452019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:38.451991 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:39.294725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:39.294697 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:39.295169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:39.295154 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:39.295940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:39.295918 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:43:41.369557 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:41.369518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:41.489070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:41.489040 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:43.439638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:43.439601 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:47.458351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:47.458288 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:43:48.314401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.314116 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k"] Nov 01 12:43:48.374638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.374499 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.401522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.401473 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.401837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.401817 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.401983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.401965 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd65d\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.402149 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.402126 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.402276 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.402261 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.402619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.402601 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.402792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.402769 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.402932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.402917 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.403144 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.403127 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.437325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.429231 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:48.551326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.509318 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.552816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.552776 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.576623 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.576533 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.576952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.576924 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.577134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.577098 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.577292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.577153 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.577568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.577536 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.577727 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.577709 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd65d\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.610355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.587728 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.610355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.543948 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.611021 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.610972 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.611295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.611275 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.612601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.612571 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.638018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.637988 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.640362 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.640334 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k"] Nov 01 12:43:48.681323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.676979 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.688972 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.688942 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.757255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.757221 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.760228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.760197 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd65d\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:48.788076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:48.788042 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:49.488933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:49.488893 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k"] Nov 01 12:43:49.560559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:49.554125 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:49.560559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:49.554220 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:49.560559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:49.554268 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:43:49.936804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:49.935973 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:49.980768 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:49.980406 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.062857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.062508 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.099072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.098503 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.099072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.098815 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:43:50.400876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.400775 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.522325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.513903 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" event={"ID":"fe467197-9c0b-4c10-aa51-6504f92be21a","Type":"ContainerStarted","Data":"9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad"} Nov 01 12:43:50.522619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.522592 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" event={"ID":"fe467197-9c0b-4c10-aa51-6504f92be21a","Type":"ContainerStarted","Data":"90f9768d5ffbbc9adfb98e38fc7fe1e9faf2bd77b10cc5aee2ff3c8cb2ad7edc"} Nov 01 12:43:50.645325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.633320 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podStartSLOduration=2.633281629 podStartE2EDuration="2.633281629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:50.628656937 +0000 UTC m=+1300.378599915" watchObservedRunningTime="2026-11-01 12:43:50.633281629 +0000 UTC m=+1300.383224608" Nov 01 12:43:50.792446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.791985 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:50.792446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.792052 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:50.792446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.792373 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:50.808100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.808057 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.820428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.819710 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.825068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.824056 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:50.981324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.980711 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 12:43:50.981324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:50.981132 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:43:51.066407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.066315 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:51.066991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.066625 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:51.067482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.067461 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:43:51.093325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.085695 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:43:51.355379 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.327813 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:51.444326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.361166 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:43:51.588323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.571918 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:43:51.717393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.709463 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 12:43:51.736626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.736590 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:43:51.738512 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.738481 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:51.820654 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.820617 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:51.821012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.820959 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:51.869743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.866419 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:51.869743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:51.868113 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:43:52.046786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.046742 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:52.067418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.067371 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:43:52.068016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.067993 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:43:52.097158 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.097128 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:43:52.137918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.137892 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:43:52.138598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.138578 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:43:52.167265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.167233 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:43:52.788739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.788709 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:52.789098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:52.788755 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:53.789382 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:53.789270 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:53.789382 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:53.789349 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:54.788731 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:54.788695 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:54.788898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:54.788756 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:54.925187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:54.925145 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:43:55.789104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:55.789065 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:55.789287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:55.789137 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:56.114059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.113983 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp"] Nov 01 12:43:56.118364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.118344 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.172415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.172379 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp"] Nov 01 12:43:56.213347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.213292 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:43:56.213347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.213333 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:43:56.272159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.272118 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf6g\" (UniqueName: \"kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.272402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.272381 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.272541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.272521 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.373467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.373377 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.373638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.373504 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.373710 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.373638 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf6g\" (UniqueName: \"kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.374126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.374104 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.375873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.375850 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.426278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.426252 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf6g\" (UniqueName: \"kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g\") pod \"maas-api-revoke-sub-33f86820-f75fp\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.432124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.432108 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:43:56.789202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.789168 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:56.789402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.789218 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:56.825230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.825203 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp"] Nov 01 12:43:56.827752 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:43:56.827721 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38424d43_d66b_468b_89c5_ac839414c91d.slice/crio-25e571c6073d1142195dd5042ea5297443f121f2a1ed2f6cc92a0361200c3687 WatchSource:0}: Error finding container 25e571c6073d1142195dd5042ea5297443f121f2a1ed2f6cc92a0361200c3687: Status 404 returned error can't find the container with id 25e571c6073d1142195dd5042ea5297443f121f2a1ed2f6cc92a0361200c3687 Nov 01 12:43:56.830452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.830435 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:43:56.993140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:56.993100 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" event={"ID":"38424d43-d66b-468b-89c5-ac839414c91d","Type":"ContainerStarted","Data":"25e571c6073d1142195dd5042ea5297443f121f2a1ed2f6cc92a0361200c3687"} Nov 01 12:43:57.789148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:57.789118 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 12:43:57.789546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:57.789163 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 12:43:58.793197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:58.793166 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:58.793577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:58.793519 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:58.794240 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:58.794221 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:43:59.927457 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:43:59.927403 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:00.256215 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:44:00.255892 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:44:02.096751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:02.096652 4484 generic.go:356] "Generic (PLEG): container finished" podID="38424d43-d66b-468b-89c5-ac839414c91d" containerID="a40b34e5f2d07b04d5acbfa4c492c9b312baf8402912e0c459da1f4153428f3b" exitCode=0 Nov 01 12:44:02.096751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:02.096729 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" event={"ID":"38424d43-d66b-468b-89c5-ac839414c91d","Type":"ContainerDied","Data":"a40b34e5f2d07b04d5acbfa4c492c9b312baf8402912e0c459da1f4153428f3b"} Nov 01 12:44:03.260832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.260810 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:44:03.301805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.301781 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g\" (UniqueName: \"kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g\") pod \"38424d43-d66b-468b-89c5-ac839414c91d\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " Nov 01 12:44:03.301937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.301859 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca\") pod \"38424d43-d66b-468b-89c5-ac839414c91d\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " Nov 01 12:44:03.301937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.301895 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca\") pod \"38424d43-d66b-468b-89c5-ac839414c91d\" (UID: \"38424d43-d66b-468b-89c5-ac839414c91d\") " Nov 01 12:44:03.302253 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.302220 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca" pod "38424d43-d66b-468b-89c5-ac839414c91d" (UID: "38424d43-d66b-468b-89c5-ac839414c91d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:03.302429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.302412 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:03.303984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.303961 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca" pod "38424d43-d66b-468b-89c5-ac839414c91d" (UID: "38424d43-d66b-468b-89c5-ac839414c91d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:44:03.304112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.304094 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g" pod "38424d43-d66b-468b-89c5-ac839414c91d" (UID: "38424d43-d66b-468b-89c5-ac839414c91d"). InnerVolumeSpecName "kube-api-access-2qf6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:03.403815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.403733 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf6g\" (UniqueName: \"kubernetes.io/projected/38424d43-d66b-468b-89c5-ac839414c91d-kube-api-access-2qf6g\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:03.403815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:03.403761 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38424d43-d66b-468b-89c5-ac839414c91d-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:04.116145 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:04.116114 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" Nov 01 12:44:04.116353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:04.116153 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp" event={"ID":"38424d43-d66b-468b-89c5-ac839414c91d","Type":"ContainerDied","Data":"25e571c6073d1142195dd5042ea5297443f121f2a1ed2f6cc92a0361200c3687"} Nov 01 12:44:04.116353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:04.116201 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e571c6073d1142195dd5042ea5297443f121f2a1ed2f6cc92a0361200c3687" Nov 01 12:44:04.950563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:04.950524 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:07.071321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:07.071264 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:07.215793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:07.215746 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:09.896270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:09.896240 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:14.573484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:14.573427 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:14.802328 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:14.801385 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:14.828393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:14.828100 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:14.850320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:14.842373 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:14.850320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:14.844402 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.030652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.030594 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.090509 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.090387 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.099431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.099375 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.144450 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.144400 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.145350 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.145218 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:44:15.170462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.170404 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.747331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.742760 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.747331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.743495 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.873379 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.873642 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.875344 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.875532 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.875675 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.875847 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.875892 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:44:15.884076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.875925 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:44:15.931338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:15.930412 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:16.049529 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.049410 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:44:16.049807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.049783 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:44:16.134325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.109197 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:16.147372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.114862 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 12:44:16.184395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.114893 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 12:44:16.185881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.150428 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:44:16.186212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.186184 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:16.212138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.212113 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:44:16.212871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.212852 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:44:16.262327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.150509 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 12:44:16.288325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.150475 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:44:16.312375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.304934 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:44:16.312375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.305015 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:44:16.475962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.451258 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.068325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.743545 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.180362 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:16.993402 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.192413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.084862 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Nov 01 12:44:17.196443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.142605 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:17.221749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.169032 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:44:17.222170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.171543 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:44:17.246135 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.171692 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:17.246732 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.171839 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:17.294326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.171959 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:44:17.294326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.214400 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.294653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.294604 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.294935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.294896 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.360384 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.314000 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:44:17.364323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.360657 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68"] Nov 01 12:44:17.420321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.369981 4484 cpu_manager.go:414] "containerMap: removing container" podUID="38424d43-d66b-468b-89c5-ac839414c91d" containerName="revoke-keys" Nov 01 12:44:17.420575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.420550 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38424d43-d66b-468b-89c5-ac839414c91d" containerName="revoke-keys" Nov 01 12:44:17.456325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.421110 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="38424d43-d66b-468b-89c5-ac839414c91d" containerName="revoke-keys" Nov 01 12:44:17.637379 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.625231 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:44:17.662325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.651911 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.707820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.707778 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:17.919375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.912040 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l"] Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954541 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954641 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954746 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954817 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954871 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954910 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.954971 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfdgp\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.955057 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:17.975337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.955085 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.070530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:17.916071 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.077523 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.078540 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.078715 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079215 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079316 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079392 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079431 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079494 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfdgp\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079599 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.079630 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.083632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.080270 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.216377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.189702 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.221090 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.201560 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.221322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.201597 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.228941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.227677 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:18.319335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.289567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.319335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.289815 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.319335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.308748 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.319335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.317650 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.319335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.319171 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.333381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.320057 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.333381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.320358 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:44:18.333381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.320609 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:18.333381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.320853 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:18.333381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.321086 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:44:18.333381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.321351 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 12:44:18.482380 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.417538 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:18.516095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.516063 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfdgp\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.641363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.621025 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.641363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.622537 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.641363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.622774 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.737678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.737252 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.737678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.737411 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.742900 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743030 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrx5\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743093 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743158 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743189 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743229 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743332 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743394 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.765325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.743573 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.867989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.867955 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.868325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.868291 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrx5\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.868497 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.868481 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.868638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.868625 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.868733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.868721 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.868838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.868826 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.868976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.868962 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.869067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.869056 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.869168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.869156 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.893747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.893536 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.899958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.899912 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:18.904378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.902720 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.904378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.903611 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.912416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.909380 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.948349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.948319 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:18.966122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.962716 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.966122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.964391 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.968823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.968381 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.970239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.970100 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.978412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.977732 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrx5\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985175 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985217 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68"] Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985272 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985291 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l"] Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985365 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985427 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985468 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985516 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985543 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:44:18.985832 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.985573 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:44:18.987639 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.986394 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:44:18.987639 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:18.986456 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:44:19.003440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.002945 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:44:19.004735 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.004715 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:44:19.018156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.018136 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:19.271451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.271366 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:19.439494 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.439466 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68"] Nov 01 12:44:19.445684 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:44:19.445655 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30058662_905d_4c70_958e_35729cf7a4ec.slice/crio-a27779c696056f6a4e1911e51cfd8735c0d411eec657dd3891eecd2bc401a6b6 WatchSource:0}: Error finding container a27779c696056f6a4e1911e51cfd8735c0d411eec657dd3891eecd2bc401a6b6: Status 404 returned error can't find the container with id a27779c696056f6a4e1911e51cfd8735c0d411eec657dd3891eecd2bc401a6b6 Nov 01 12:44:19.448281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.448243 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:19.448528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.448343 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:19.448528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.448393 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:19.666957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.666916 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l"] Nov 01 12:44:19.673431 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:44:19.673400 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a8fe3d_eea9_4247_a45b_b9d0f3da3255.slice/crio-53c715b50fa624a0df4f40778ac50e1c869ac52b802595682fd870242b5747bc WatchSource:0}: Error finding container 53c715b50fa624a0df4f40778ac50e1c869ac52b802595682fd870242b5747bc: Status 404 returned error can't find the container with id 53c715b50fa624a0df4f40778ac50e1c869ac52b802595682fd870242b5747bc Nov 01 12:44:19.676893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.676856 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:19.676978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.676940 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:19.677022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.676986 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:19.972901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.972811 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" event={"ID":"30058662-905d-4c70-958e-35729cf7a4ec","Type":"ContainerStarted","Data":"be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7"} Nov 01 12:44:19.973056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.972906 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" event={"ID":"30058662-905d-4c70-958e-35729cf7a4ec","Type":"ContainerStarted","Data":"a27779c696056f6a4e1911e51cfd8735c0d411eec657dd3891eecd2bc401a6b6"} Nov 01 12:44:19.974915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.974878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" event={"ID":"70a8fe3d-eea9-4247-a45b-b9d0f3da3255","Type":"ContainerStarted","Data":"f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640"} Nov 01 12:44:19.975031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:19.974929 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" event={"ID":"70a8fe3d-eea9-4247-a45b-b9d0f3da3255","Type":"ContainerStarted","Data":"53c715b50fa624a0df4f40778ac50e1c869ac52b802595682fd870242b5747bc"} Nov 01 12:44:20.281006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.280972 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:20.282680 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.282654 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:20.282878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.282851 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:20.735358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.735255 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podStartSLOduration=4.735235279 podStartE2EDuration="4.735235279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:20.733503548 +0000 UTC m=+1330.483446527" watchObservedRunningTime="2026-11-01 12:44:20.735235279 +0000 UTC m=+1330.485178260" Nov 01 12:44:20.887382 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.887294 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podStartSLOduration=4.887267846 podStartE2EDuration="4.887267846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:20.860629359 +0000 UTC m=+1330.610572339" watchObservedRunningTime="2026-11-01 12:44:20.887267846 +0000 UTC m=+1330.637210825" Nov 01 12:44:20.971330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.970365 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:20.971330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.970484 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 12:44:20.971330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:20.970528 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 12:44:21.290369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:21.275627 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:21.290369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:21.275688 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:21.968249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:21.968212 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 12:44:21.968558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:21.968529 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 12:44:22.292756 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:22.291756 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:22.292756 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:22.291818 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:22.984066 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:22.983423 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 12:44:22.984066 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:22.983486 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 12:44:23.274757 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:23.274663 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:23.275030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:23.274994 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:23.988236 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:23.988201 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 12:44:23.988827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:23.988797 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 12:44:24.272882 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:24.272787 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:24.273151 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:24.273122 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:24.974324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:24.973544 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 12:44:24.974324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:24.973616 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 12:44:25.117511 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:25.117479 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:25.273758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:25.273720 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:25.274057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:25.274027 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:25.967661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:25.967621 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 12:44:25.969674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:25.969642 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 12:44:26.271924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:26.271892 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 12:44:26.272347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:26.271946 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 12:44:27.126094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:27.123819 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:27.126094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:27.124596 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:27.126094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:27.125597 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:44:27.510120 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:27.510086 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:27.513468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:27.513444 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:27.524944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:27.524915 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:44:30.251714 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:44:30.251675 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:44:30.707745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:30.707668 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:31.468896 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.468865 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp"] Nov 01 12:44:31.469397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.469075 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" containerID="cri-o://5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5" gracePeriod=30 Nov 01 12:44:31.524248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.524209 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf"] Nov 01 12:44:31.524816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.524762 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" containerID="cri-o://30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db" gracePeriod=30 Nov 01 12:44:31.526986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.526959 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6"] Nov 01 12:44:31.527191 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.527146 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" containerID="cri-o://8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41" gracePeriod=30 Nov 01 12:44:31.561240 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.561211 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj"] Nov 01 12:44:31.561429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.561412 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" containerID="cri-o://6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344" gracePeriod=30 Nov 01 12:44:31.564681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.564656 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8"] Nov 01 12:44:31.568816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.568798 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691191 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691255 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691285 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691367 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691441 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcp2\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691490 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691638 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691691 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.694321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.691803 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.697824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.696710 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8"] Nov 01 12:44:31.792851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.792819 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.792883 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.792953 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793000 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793028 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793053 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793208 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793245 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793346 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcp2\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793404 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793429 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793454 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793531 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.793844 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.793821 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.795426 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.795405 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.795813 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.795793 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.845775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.845751 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.845888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.845837 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcp2\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:31.888500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:31.888475 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:32.316873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:32.316842 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8"] Nov 01 12:44:32.323189 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:44:32.323162 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2d570e_41b7_429b_9d41_9ffc4e47dd73.slice/crio-dd9cbfd8bd6bad54e50e5f92f7e737790b46a0b15ea351be54890f63a2c152f4 WatchSource:0}: Error finding container dd9cbfd8bd6bad54e50e5f92f7e737790b46a0b15ea351be54890f63a2c152f4: Status 404 returned error can't find the container with id dd9cbfd8bd6bad54e50e5f92f7e737790b46a0b15ea351be54890f63a2c152f4 Nov 01 12:44:32.326261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:32.326227 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:32.326374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:32.326308 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:32.326374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:32.326338 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:44:33.086553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:33.086521 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" event={"ID":"6b2d570e-41b7-429b-9d41-9ffc4e47dd73","Type":"ContainerStarted","Data":"c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1"} Nov 01 12:44:33.087002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:33.086574 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" event={"ID":"6b2d570e-41b7-429b-9d41-9ffc4e47dd73","Type":"ContainerStarted","Data":"dd9cbfd8bd6bad54e50e5f92f7e737790b46a0b15ea351be54890f63a2c152f4"} Nov 01 12:44:33.199067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:33.199014 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podStartSLOduration=2.198999241 podStartE2EDuration="2.198999241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:33.196062928 +0000 UTC m=+1342.946005906" watchObservedRunningTime="2026-11-01 12:44:33.198999241 +0000 UTC m=+1342.948942218" Nov 01 12:44:33.889640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:33.889610 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:33.891349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:33.891294 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 12:44:33.891482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:33.891377 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 12:44:34.522197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.522167 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j"] Nov 01 12:44:34.534170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.534143 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.579199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.579173 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j"] Nov 01 12:44:34.621269 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.621239 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:44:34.621269 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.621265 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:44:34.636053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.636024 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.636267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.636247 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87t4\" (UniqueName: \"kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.636338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.636278 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.737682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.737653 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.737899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.737879 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87t4\" (UniqueName: \"kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.737953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.737917 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.738701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.738672 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.740010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.739983 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.793913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.793842 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87t4\" (UniqueName: \"kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4\") pod \"maas-api-revoke-sub-9c3440bb-5kj9j\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.873200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.873166 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:34.889153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.889127 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 12:44:34.889268 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:34.889172 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 12:44:35.090940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.090865 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-szcjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:44:35.090940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.090916 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:44:35.268526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.268485 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j"] Nov 01 12:44:35.274349 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:44:35.272891 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec7e532_35e8_4450_8006_9494ab757de7.slice/crio-e25517ab042ca1b44c4bf2e229881464ad6e8cdce59bc182628cd5ce77a14c5a WatchSource:0}: Error finding container e25517ab042ca1b44c4bf2e229881464ad6e8cdce59bc182628cd5ce77a14c5a: Status 404 returned error can't find the container with id e25517ab042ca1b44c4bf2e229881464ad6e8cdce59bc182628cd5ce77a14c5a Nov 01 12:44:35.483971 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.483939 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-bzhdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:44:35.484119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.484002 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:44:35.889340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.889293 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 12:44:35.889720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:35.889360 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 12:44:36.110952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.110863 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" event={"ID":"dec7e532-35e8-4450-8006-9494ab757de7","Type":"ContainerStarted","Data":"0a71d574af485402bb6b22b015ce94a0c4938d7741afd9e3d0546d129f8e481e"} Nov 01 12:44:36.110952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.110919 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" event={"ID":"dec7e532-35e8-4450-8006-9494ab757de7","Type":"ContainerStarted","Data":"e25517ab042ca1b44c4bf2e229881464ad6e8cdce59bc182628cd5ce77a14c5a"} Nov 01 12:44:36.218737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.218688 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" podStartSLOduration=2.218672509 podStartE2EDuration="2.218672509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:36.218403192 +0000 UTC m=+1345.968346167" watchObservedRunningTime="2026-11-01 12:44:36.218672509 +0000 UTC m=+1345.968615486" Nov 01 12:44:36.339412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.339380 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:36.889195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.889127 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 12:44:36.889335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.889190 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 12:44:36.905273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.905255 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:44:36.998565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.998533 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:36.998941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.998922 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:36.999141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.999124 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:36.999351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.999328 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:36.999506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.999481 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:36.999723 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.999705 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:36.999921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:36.999904 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:37.000065 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.000047 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:37.000176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.000161 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket\") pod \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\" (UID: \"39a531eb-ae89-4ab2-9aac-029c6da7c74e\") " Nov 01 12:44:37.000415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.000082 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.000630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.000590 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.000764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.000703 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.001057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.001016 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.001262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.001134 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.001262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.001203 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.001262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.001222 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.002834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.002799 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:37.003283 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.003245 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:44:37.004332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.004097 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.008742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.008720 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "kube-api-access-8zlcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.008963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.008916 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy" pod "39a531eb-ae89-4ab2-9aac-029c6da7c74e" (UID: "39a531eb-ae89-4ab2-9aac-029c6da7c74e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.085313 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.085270 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:44:37.087407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.087383 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:44:37.089410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.089391 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:44:37.103129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.103013 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.103129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.103045 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.103129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.103062 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.103129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.103076 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.103129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.103092 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlcl\" (UniqueName: \"kubernetes.io/projected/39a531eb-ae89-4ab2-9aac-029c6da7c74e-kube-api-access-8zlcl\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.103129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.103106 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a531eb-ae89-4ab2-9aac-029c6da7c74e-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.125988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.125956 4484 generic.go:356] "Generic (PLEG): container finished" podID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerID="5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5" exitCode=0 Nov 01 12:44:37.126111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.126017 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" Nov 01 12:44:37.126111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.126041 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" event={"ID":"39a531eb-ae89-4ab2-9aac-029c6da7c74e","Type":"ContainerDied","Data":"5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5"} Nov 01 12:44:37.126213 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.126126 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp" event={"ID":"39a531eb-ae89-4ab2-9aac-029c6da7c74e","Type":"ContainerDied","Data":"45f5e6829f15c90fddd8609f1cf0e9f06fa8db1a66df8e95239f62d77d528ca4"} Nov 01 12:44:37.126213 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.126155 4484 scope.go:122] "RemoveContainer" containerID="5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5" Nov 01 12:44:37.127462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.127436 4484 generic.go:356] "Generic (PLEG): container finished" podID="0cdc3920-884a-4987-a830-bc24964bc85f" containerID="8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41" exitCode=0 Nov 01 12:44:37.127569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.127503 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" Nov 01 12:44:37.127569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.127523 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" event={"ID":"0cdc3920-884a-4987-a830-bc24964bc85f","Type":"ContainerDied","Data":"8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41"} Nov 01 12:44:37.127686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.127589 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6" event={"ID":"0cdc3920-884a-4987-a830-bc24964bc85f","Type":"ContainerDied","Data":"03f46e4d4ced9377441a10b14036dc13afa588b61f14dc302e53305af1e7d23a"} Nov 01 12:44:37.129002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.128984 4484 generic.go:356] "Generic (PLEG): container finished" podID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerID="6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344" exitCode=0 Nov 01 12:44:37.129084 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.129046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" event={"ID":"e49ef56a-ea03-49cb-87a6-9c277eb01420","Type":"ContainerDied","Data":"6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344"} Nov 01 12:44:37.129084 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.129074 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" event={"ID":"e49ef56a-ea03-49cb-87a6-9c277eb01420","Type":"ContainerDied","Data":"0f6846ba3495f6d6bb087d0e803dd5ab763a02c24fd5d592c7b7df65d6a5360e"} Nov 01 12:44:37.129170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.129047 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj" Nov 01 12:44:37.130438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.130419 4484 generic.go:356] "Generic (PLEG): container finished" podID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerID="30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db" exitCode=0 Nov 01 12:44:37.130538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.130478 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" Nov 01 12:44:37.130538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.130495 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" event={"ID":"2b76a5a1-fa75-4b8e-ae12-64195448c023","Type":"ContainerDied","Data":"30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db"} Nov 01 12:44:37.130538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.130528 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf" event={"ID":"2b76a5a1-fa75-4b8e-ae12-64195448c023","Type":"ContainerDied","Data":"126205c3faabb4c83531c7a86ad47025d401cbc49bdfbffae6b443cb1736feab"} Nov 01 12:44:37.135749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.132460 4484 generic.go:356] "Generic (PLEG): container finished" podID="dec7e532-35e8-4450-8006-9494ab757de7" containerID="0a71d574af485402bb6b22b015ce94a0c4938d7741afd9e3d0546d129f8e481e" exitCode=0 Nov 01 12:44:37.135749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.132539 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" event={"ID":"dec7e532-35e8-4450-8006-9494ab757de7","Type":"ContainerDied","Data":"0a71d574af485402bb6b22b015ce94a0c4938d7741afd9e3d0546d129f8e481e"} Nov 01 12:44:37.141204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.141184 4484 scope.go:122] "RemoveContainer" containerID="5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5" Nov 01 12:44:37.149324 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:44:37.146437 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5\": container with ID starting with 5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5 not found: ID does not exist" containerID="5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5" Nov 01 12:44:37.149324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.146476 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5"} err="failed to get container status \"5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5\": rpc error: code = NotFound desc = could not find container \"5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5\": container with ID starting with 5174b88ffd671ed4ca4d31e02f25cab5631458bd4fa8f15a7e34b6395bb583d5 not found: ID does not exist" Nov 01 12:44:37.149324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.146499 4484 scope.go:122] "RemoveContainer" containerID="8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41" Nov 01 12:44:37.172513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.162903 4484 scope.go:122] "RemoveContainer" containerID="8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41" Nov 01 12:44:37.172513 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:44:37.163272 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41\": container with ID starting with 8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41 not found: ID does not exist" containerID="8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41" Nov 01 12:44:37.172513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.163372 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41"} err="failed to get container status \"8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41\": rpc error: code = NotFound desc = could not find container \"8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41\": container with ID starting with 8b939fd6790e868afa12910b74b6eaf6d769cbb4ac587092daabd9081cef4a41 not found: ID does not exist" Nov 01 12:44:37.172513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.163395 4484 scope.go:122] "RemoveContainer" containerID="6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344" Nov 01 12:44:37.184017 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.183992 4484 scope.go:122] "RemoveContainer" containerID="6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344" Nov 01 12:44:37.184285 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:44:37.184265 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344\": container with ID starting with 6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344 not found: ID does not exist" containerID="6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344" Nov 01 12:44:37.184389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.184291 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344"} err="failed to get container status \"6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344\": rpc error: code = NotFound desc = could not find container \"6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344\": container with ID starting with 6a007705402bb946fec9ac17af9fce2b65359169ae42be5bfa254525e1039344 not found: ID does not exist" Nov 01 12:44:37.184389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.184323 4484 scope.go:122] "RemoveContainer" containerID="30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db" Nov 01 12:44:37.194064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.194043 4484 scope.go:122] "RemoveContainer" containerID="30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db" Nov 01 12:44:37.194371 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:44:37.194342 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db\": container with ID starting with 30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db not found: ID does not exist" containerID="30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db" Nov 01 12:44:37.194453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.194382 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db"} err="failed to get container status \"30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db\": rpc error: code = NotFound desc = could not find container \"30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db\": container with ID starting with 30edb19fd462a793f2d0a718c1e11712c0270b36615e1c18d1504e6b5ccc56db not found: ID does not exist" Nov 01 12:44:37.203761 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203737 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.203875 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203775 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.203875 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203804 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.203875 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203838 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.204046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203902 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.204046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203945 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.204046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.203989 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.204198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204057 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.204198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204075 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.204198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204096 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.204365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204203 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.204365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204229 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.204365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204265 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.204365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204313 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.204365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204342 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.204619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204526 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.204619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204600 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.204892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204871 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.205443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.204919 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:37.205443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.205023 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.205443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.205228 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.205992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.205768 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.205992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.205953 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:37.206171 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.206153 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.206675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.206642 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.206779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.206764 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.206944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.206926 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.207422 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.207331 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:37.207538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.207517 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:44:37.207661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.207642 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.207919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.207807 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.208075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208058 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.208207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.207901 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.208282 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208186 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.208282 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208253 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.208440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208338 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket\") pod \"e49ef56a-ea03-49cb-87a6-9c277eb01420\" (UID: \"e49ef56a-ea03-49cb-87a6-9c277eb01420\") " Nov 01 12:44:37.208440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data\") pod \"2b76a5a1-fa75-4b8e-ae12-64195448c023\" (UID: \"2b76a5a1-fa75-4b8e-ae12-64195448c023\") " Nov 01 12:44:37.208440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208416 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.208608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208474 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data\") pod \"0cdc3920-884a-4987-a830-bc24964bc85f\" (UID: \"0cdc3920-884a-4987-a830-bc24964bc85f\") " Nov 01 12:44:37.209027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.208873 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.209115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.209068 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.209575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.209554 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ef56a-ea03-49cb-87a6-9c277eb01420-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.209692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.209389 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.209797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.209780 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.209888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.209875 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210020 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210049 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210064 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210077 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdc3920-884a-4987-a830-bc24964bc85f-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210092 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210108 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b76a5a1-fa75-4b8e-ae12-64195448c023-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210121 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210135 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210148 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210162 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.210235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.210178 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdc3920-884a-4987-a830-bc24964bc85f-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.214909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.214825 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "kube-api-access-sv86q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.215177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.215149 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.216584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.216539 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.221605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.218915 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:44:37.221859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.219025 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.221952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.221875 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:44:37.222916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.222882 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.223015 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.222930 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.223234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.223189 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:44:37.223507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.223470 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr" pod "0cdc3920-884a-4987-a830-bc24964bc85f" (UID: "0cdc3920-884a-4987-a830-bc24964bc85f"). InnerVolumeSpecName "kube-api-access-2xjkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.223692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.223663 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token" pod "e49ef56a-ea03-49cb-87a6-9c277eb01420" (UID: "e49ef56a-ea03-49cb-87a6-9c277eb01420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.225559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.225530 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57" pod "2b76a5a1-fa75-4b8e-ae12-64195448c023" (UID: "2b76a5a1-fa75-4b8e-ae12-64195448c023"). InnerVolumeSpecName "kube-api-access-97t57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:37.311198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311166 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311196 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311207 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv86q\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-kube-api-access-sv86q\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311218 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311227 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311235 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311244 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ef56a-ea03-49cb-87a6-9c277eb01420-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311252 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjkr\" (UniqueName: \"kubernetes.io/projected/0cdc3920-884a-4987-a830-bc24964bc85f-kube-api-access-2xjkr\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311260 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311269 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdc3920-884a-4987-a830-bc24964bc85f-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311279 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311288 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.311416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.311296 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97t57\" (UniqueName: \"kubernetes.io/projected/2b76a5a1-fa75-4b8e-ae12-64195448c023-kube-api-access-97t57\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:37.386063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.386033 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp"] Nov 01 12:44:37.389456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.389427 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sdqqp"] Nov 01 12:44:37.829461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.829426 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf"] Nov 01 12:44:37.833823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.833798 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-bzhdf"] Nov 01 12:44:37.889782 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.889755 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 12:44:37.889891 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.889805 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 12:44:37.938412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.938382 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj"] Nov 01 12:44:37.939242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:37.939190 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-szcjj"] Nov 01 12:44:38.038330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.038287 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6"] Nov 01 12:44:38.068456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.068419 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-7gzn6"] Nov 01 12:44:38.341562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.341531 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:38.370833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.370804 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" path="/var/lib/kubelet/pods/0cdc3920-884a-4987-a830-bc24964bc85f/volumes" Nov 01 12:44:38.371541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.371514 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" path="/var/lib/kubelet/pods/2b76a5a1-fa75-4b8e-ae12-64195448c023/volumes" Nov 01 12:44:38.372181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.372159 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" path="/var/lib/kubelet/pods/39a531eb-ae89-4ab2-9aac-029c6da7c74e/volumes" Nov 01 12:44:38.373215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.373196 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" path="/var/lib/kubelet/pods/e49ef56a-ea03-49cb-87a6-9c277eb01420/volumes" Nov 01 12:44:38.430041 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.429985 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca\") pod \"dec7e532-35e8-4450-8006-9494ab757de7\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " Nov 01 12:44:38.430041 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.430024 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4\" (UniqueName: \"kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4\") pod \"dec7e532-35e8-4450-8006-9494ab757de7\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " Nov 01 12:44:38.430225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.430072 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca\") pod \"dec7e532-35e8-4450-8006-9494ab757de7\" (UID: \"dec7e532-35e8-4450-8006-9494ab757de7\") " Nov 01 12:44:38.430615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.430582 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca" pod "dec7e532-35e8-4450-8006-9494ab757de7" (UID: "dec7e532-35e8-4450-8006-9494ab757de7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:38.430983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.430959 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:38.432272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.432253 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca" pod "dec7e532-35e8-4450-8006-9494ab757de7" (UID: "dec7e532-35e8-4450-8006-9494ab757de7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:44:38.432272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.432260 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4" pod "dec7e532-35e8-4450-8006-9494ab757de7" (UID: "dec7e532-35e8-4450-8006-9494ab757de7"). InnerVolumeSpecName "kube-api-access-p87t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:38.532821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.532790 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec7e532-35e8-4450-8006-9494ab757de7-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:38.532821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.532818 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87t4\" (UniqueName: \"kubernetes.io/projected/dec7e532-35e8-4450-8006-9494ab757de7-kube-api-access-p87t4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:38.889225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.889192 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 12:44:38.889389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:38.889266 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 12:44:39.174814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:39.174716 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" event={"ID":"dec7e532-35e8-4450-8006-9494ab757de7","Type":"ContainerDied","Data":"e25517ab042ca1b44c4bf2e229881464ad6e8cdce59bc182628cd5ce77a14c5a"} Nov 01 12:44:39.174814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:39.174765 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25517ab042ca1b44c4bf2e229881464ad6e8cdce59bc182628cd5ce77a14c5a" Nov 01 12:44:39.174814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:39.174778 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j" Nov 01 12:44:39.893269 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:39.893243 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:39.893740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:39.893724 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:39.894389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:39.894368 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:44:41.806393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:41.806360 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:44.733612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:44.733580 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:44.856698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:44.856671 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:45.056019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:45.055930 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:47.201012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:47.200969 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:44:48.167926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.167896 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk"] Nov 01 12:44:48.168486 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168472 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dec7e532-35e8-4450-8006-9494ab757de7" containerName="revoke-keys" Nov 01 12:44:48.168551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168486 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec7e532-35e8-4450-8006-9494ab757de7" containerName="revoke-keys" Nov 01 12:44:48.168551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168508 4484 cpu_manager.go:414] "containerMap: removing container" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" Nov 01 12:44:48.168551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168513 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" Nov 01 12:44:48.168551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168529 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" Nov 01 12:44:48.168551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168534 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" Nov 01 12:44:48.168551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168553 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168557 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168564 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168568 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168651 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a531eb-ae89-4ab2-9aac-029c6da7c74e" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168660 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b76a5a1-fa75-4b8e-ae12-64195448c023" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168671 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdc3920-884a-4987-a830-bc24964bc85f" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168676 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49ef56a-ea03-49cb-87a6-9c277eb01420" containerName="istio-proxy" Nov 01 12:44:48.168725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.168683 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec7e532-35e8-4450-8006-9494ab757de7" containerName="revoke-keys" Nov 01 12:44:48.173567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.173547 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.242835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.242799 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk"] Nov 01 12:44:48.267929 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.267897 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:44:48.268122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.267935 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:44:48.291515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.291482 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwwb\" (UniqueName: \"kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.291624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.291537 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.291788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.291763 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.392759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.392733 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwwb\" (UniqueName: \"kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.392895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.392778 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.392963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.392923 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.393429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.393408 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.395079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.395061 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.447372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.447315 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwwb\" (UniqueName: \"kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb\") pod \"maas-api-revoke-sub-0e9cbc73-9gmhk\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.487536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.487510 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:48.591670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.591635 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:44:48.870791 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:48.870750 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk"] Nov 01 12:44:48.884366 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:44:48.882941 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee95598_ed34_425f_a0c1_f1ab511886fc.slice/crio-c9d70c17e889b3bd18709e0f0548da9056b92e1fed323bb00ff12e6dbf7ac3c0 WatchSource:0}: Error finding container c9d70c17e889b3bd18709e0f0548da9056b92e1fed323bb00ff12e6dbf7ac3c0: Status 404 returned error can't find the container with id c9d70c17e889b3bd18709e0f0548da9056b92e1fed323bb00ff12e6dbf7ac3c0 Nov 01 12:44:49.260200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:49.260162 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" event={"ID":"7ee95598-ed34-425f-a0c1-f1ab511886fc","Type":"ContainerStarted","Data":"c9d70c17e889b3bd18709e0f0548da9056b92e1fed323bb00ff12e6dbf7ac3c0"} Nov 01 12:44:50.433383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.427502 4484 generic.go:356] "Generic (PLEG): container finished" podID="7ee95598-ed34-425f-a0c1-f1ab511886fc" containerID="3fa86656988ff8ef57dac0c72f6a35a5238a07b487a87b57d8147addf227dab6" exitCode=0 Nov 01 12:44:50.438328 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.435066 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" event={"ID":"7ee95598-ed34-425f-a0c1-f1ab511886fc","Type":"ContainerDied","Data":"3fa86656988ff8ef57dac0c72f6a35a5238a07b487a87b57d8147addf227dab6"} Nov 01 12:44:50.678754 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.678046 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:44:50.917326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.913883 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:50.917326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.916905 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:50.964326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.964146 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:50.977199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:50.977147 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.008624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.008584 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.254323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.253450 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.254323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.253959 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:51.340321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.337431 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 12:44:51.340321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.337897 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 12:44:51.340321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.339850 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:44:51.340321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.340177 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:44:51.395942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.395899 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.601386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.589867 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.796331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.796266 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.797043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.797018 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:44:51.958420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.958246 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.973751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.968842 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:44:51.973751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.970081 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:51.973751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.971275 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:51.973751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.972468 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:44:51.973751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.972523 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:44:51.989335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.988206 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:44:51.991828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:51.989588 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:44:52.025935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.025896 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:52.028014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.027989 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:44:52.252678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.252646 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:52.266690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.266648 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:52.266974 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.266947 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:44:52.269519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.269492 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:44:52.270411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.270385 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:52.280323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.272612 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:44:52.360529 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.360480 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:52.369373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.362645 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:44:52.634353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.633971 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:52.664327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.636794 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:44:52.704427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.640535 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:52.776399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.760668 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:44:52.900322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.896275 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:52.900915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.900875 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:52.924909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.924873 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:44:52.927094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:52.927073 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:44:53.142195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.142095 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.142582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.142550 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.208080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.175373 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.208588 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.175402 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 12:44:53.209078 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.177094 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:53.255940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.255899 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:53.288719 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.288680 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.289107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.289085 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:44:53.363143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.363101 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.448778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.448680 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:53.449136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.449089 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.449361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.448984 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:53.467767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.467735 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca\") pod \"7ee95598-ed34-425f-a0c1-f1ab511886fc\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " Nov 01 12:44:53.468115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.468081 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb\" (UniqueName: \"kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb\") pod \"7ee95598-ed34-425f-a0c1-f1ab511886fc\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " Nov 01 12:44:53.468467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.468432 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca\") pod \"7ee95598-ed34-425f-a0c1-f1ab511886fc\" (UID: \"7ee95598-ed34-425f-a0c1-f1ab511886fc\") " Nov 01 12:44:53.468647 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.468628 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca" pod "7ee95598-ed34-425f-a0c1-f1ab511886fc" (UID: "7ee95598-ed34-425f-a0c1-f1ab511886fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:44:53.469825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.469799 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:53.613326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.596699 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.654358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.622516 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb" pod "7ee95598-ed34-425f-a0c1-f1ab511886fc" (UID: "7ee95598-ed34-425f-a0c1-f1ab511886fc"). InnerVolumeSpecName "kube-api-access-bpwwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:44:53.654358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.629857 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca" pod "7ee95598-ed34-425f-a0c1-f1ab511886fc" (UID: "7ee95598-ed34-425f-a0c1-f1ab511886fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:44:53.681327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.661916 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:53.749371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.744383 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ee95598-ed34-425f-a0c1-f1ab511886fc-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:53.749371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.744426 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpwwb\" (UniqueName: \"kubernetes.io/projected/7ee95598-ed34-425f-a0c1-f1ab511886fc-kube-api-access-bpwwb\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:44:53.808628 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:53.799148 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.466381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.360241 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.382408 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.400479 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.432683 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.436645 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.436679 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.436701 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.436730 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:54.824327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.437140 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.479546 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.862447 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.700212 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" event={"ID":"7ee95598-ed34-425f-a0c1-f1ab511886fc","Type":"ContainerDied","Data":"c9d70c17e889b3bd18709e0f0548da9056b92e1fed323bb00ff12e6dbf7ac3c0"} Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.862550 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d70c17e889b3bd18709e0f0548da9056b92e1fed323bb00ff12e6dbf7ac3c0" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.706634 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.597541 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.519410 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.863204 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.777424 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.863263 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:54.863326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.700292 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk" Nov 01 12:44:54.906327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:54.815947 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.435378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.434483 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:55.435378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.434549 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.455121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.453840 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:55.455121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.453909 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.844180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.843030 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.857685 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.862376 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.862668 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.862858 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.863020 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.863172 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.863899 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.864110 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.864285 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.864474 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:55.891344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:55.887489 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:56.019646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.019507 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:44:56.019646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.019573 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:56.109893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.109777 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 12:44:56.638484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.586495 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": context deadline exceeded" Nov 01 12:44:56.860742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.860701 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:44:56.937338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.937209 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:44:56.998530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:56.998489 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:57.010716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.010663 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:44:57.012912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.012868 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 12:44:57.053866 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.049174 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:44:57.073071 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.073015 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 12:44:57.089687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.089652 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 12:44:57.091630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.088374 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded" Nov 01 12:44:57.106836 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.106786 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:44:57.220783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.194278 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:44:57.220783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.194447 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 12:44:57.220783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.194646 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 12:44:57.272873 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.272845 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" Nov 01 12:44:57.327576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.327134 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 12:44:57.380323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.377669 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 12:44:57.404774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.404749 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:44:57.419377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.410508 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:44:57.422220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.421342 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:44:57.424323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.424280 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:44:57.425217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.425191 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 12:44:57.426953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.426939 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:44:57.429455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.429441 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:44:57.434269 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:57.431718 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:44:58.877880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:44:58.877850 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:00.192699 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.192660 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt"] Nov 01 12:45:00.193350 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.193331 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7ee95598-ed34-425f-a0c1-f1ab511886fc" containerName="revoke-keys" Nov 01 12:45:00.193411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.193353 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee95598-ed34-425f-a0c1-f1ab511886fc" containerName="revoke-keys" Nov 01 12:45:00.193507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.193497 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee95598-ed34-425f-a0c1-f1ab511886fc" containerName="revoke-keys" Nov 01 12:45:00.197599 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.197577 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d"] Nov 01 12:45:00.197730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.197697 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.201800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.201777 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.241217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.241183 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c"] Nov 01 12:45:00.246456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.246428 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.263523 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:00.263426 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:45:00.281766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.281727 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:45:00.285828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.285798 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt"] Nov 01 12:45:00.291085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.291034 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.291296 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.291160 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.291296 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.291188 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mfv\" (UniqueName: \"kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv\") pod \"maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.291296 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.291253 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmpx\" (UniqueName: \"kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.301512 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.301466 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d"] Nov 01 12:45:00.345062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.345026 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c"] Nov 01 12:45:00.392645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.392609 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.392871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.392753 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.392938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.392875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.392938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.392912 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mfv\" (UniqueName: \"kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv\") pod \"maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.393069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.393002 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmpx\" (UniqueName: \"kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.393126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.393097 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65f9j\" (UniqueName: \"kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.393544 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.393520 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.393757 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.393739 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.447593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.447525 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mfv\" (UniqueName: \"kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv\") pod \"maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.447593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.447534 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmpx\" (UniqueName: \"kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.494938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.494903 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65f9j\" (UniqueName: \"kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.495112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.495036 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.495590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.495572 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.513721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.513701 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:00.523032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.523013 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:00.550864 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.550810 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65f9j\" (UniqueName: \"kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.566221 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.566186 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:00.916767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.911609 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt"] Nov 01 12:45:00.965021 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.964617 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c"] Nov 01 12:45:00.969855 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:00.969821 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d"] Nov 01 12:45:00.970225 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:00.970197 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0a352f_a9ff_413e_be1b_fb3a6001fc8d.slice/crio-cf1a6cf74f8a6aca569767a0427ae93f730e09daaf9345d4997866ce33abfe6f WatchSource:0}: Error finding container cf1a6cf74f8a6aca569767a0427ae93f730e09daaf9345d4997866ce33abfe6f: Status 404 returned error can't find the container with id cf1a6cf74f8a6aca569767a0427ae93f730e09daaf9345d4997866ce33abfe6f Nov 01 12:45:00.972055 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:00.972030 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf281d63b_66e2_49b1_bb8c_1c4d96b566d5.slice/crio-454810bc6e2b0c03738353ee4f31e9d1b07798876484df38b9283f9b3c3aafac WatchSource:0}: Error finding container 454810bc6e2b0c03738353ee4f31e9d1b07798876484df38b9283f9b3c3aafac: Status 404 returned error can't find the container with id 454810bc6e2b0c03738353ee4f31e9d1b07798876484df38b9283f9b3c3aafac Nov 01 12:45:01.296100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.296056 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" event={"ID":"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08","Type":"ContainerStarted","Data":"3194de6c8537583f4acd60dbc53fe2e8dc0a0f718d8b0215788c4ec2aeb71c35"} Nov 01 12:45:01.297570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.297543 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" event={"ID":"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d","Type":"ContainerStarted","Data":"cf1a6cf74f8a6aca569767a0427ae93f730e09daaf9345d4997866ce33abfe6f"} Nov 01 12:45:01.298958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.298935 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerStarted","Data":"ad927d4054b6fd948547471224f6bf7267e0226a29e91b3af8fe48fad6138a04"} Nov 01 12:45:01.299086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.298980 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerStarted","Data":"454810bc6e2b0c03738353ee4f31e9d1b07798876484df38b9283f9b3c3aafac"} Nov 01 12:45:01.307781 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.307759 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:01.414291 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.414246 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" podStartSLOduration=1.414233373 podStartE2EDuration="1.414233373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:01.414067042 +0000 UTC m=+1371.164010016" watchObservedRunningTime="2026-11-01 12:45:01.414233373 +0000 UTC m=+1371.164176345" Nov 01 12:45:01.515608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.515537 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:01.698125 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.698092 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:01.901932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:01.901893 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:02.156399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.156174 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:02.270848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.270609 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h"] Nov 01 12:45:02.275544 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.275519 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.337338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.336673 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:45:02.440798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.376977 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:02.440798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.431363 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.440798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.431517 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdxq\" (UniqueName: \"kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.440798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.431733 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.525324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.522976 4484 generic.go:356] "Generic (PLEG): container finished" podID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerID="058afc9bce2195d3a79ee2944b3b020641634da955002ede06e3eebb16efcf02" exitCode=60 Nov 01 12:45:02.525324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.523492 4484 scope.go:122] "RemoveContainer" containerID="058afc9bce2195d3a79ee2944b3b020641634da955002ede06e3eebb16efcf02" Nov 01 12:45:02.560502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.534638 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.560502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.534737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.560502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.534842 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdxq\" (UniqueName: \"kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.560502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.541752 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.560502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.542583 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.772323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.771579 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdxq\" (UniqueName: \"kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq\") pod \"maas-api-revoke-sub-814980d2-cp87h\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:02.776320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.773183 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h"] Nov 01 12:45:02.776320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.773255 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" event={"ID":"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d","Type":"ContainerDied","Data":"058afc9bce2195d3a79ee2944b3b020641634da955002ede06e3eebb16efcf02"} Nov 01 12:45:02.785321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.782267 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:02.785321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.783651 4484 generic.go:356] "Generic (PLEG): container finished" podID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerID="ad927d4054b6fd948547471224f6bf7267e0226a29e91b3af8fe48fad6138a04" exitCode=60 Nov 01 12:45:02.785321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.783797 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerDied","Data":"ad927d4054b6fd948547471224f6bf7267e0226a29e91b3af8fe48fad6138a04"} Nov 01 12:45:02.785321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.784263 4484 scope.go:122] "RemoveContainer" containerID="ad927d4054b6fd948547471224f6bf7267e0226a29e91b3af8fe48fad6138a04" Nov 01 12:45:02.789321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.786816 4484 generic.go:356] "Generic (PLEG): container finished" podID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerID="a6277b45be5e75002095c50844137520ab995808d615f7bbc6c694cf0221b290" exitCode=60 Nov 01 12:45:02.789321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.787030 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" event={"ID":"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08","Type":"ContainerDied","Data":"a6277b45be5e75002095c50844137520ab995808d615f7bbc6c694cf0221b290"} Nov 01 12:45:02.789321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.787477 4484 scope.go:122] "RemoveContainer" containerID="a6277b45be5e75002095c50844137520ab995808d615f7bbc6c694cf0221b290" Nov 01 12:45:02.899128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:02.899099 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:03.293410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.293374 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h"] Nov 01 12:45:03.297410 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:03.297379 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf8256d_ade3_4ce8_be64_404de4d0fc46.slice/crio-852e38394cd620480ee4f36fc7db4d4e260c01e31bd665e9837da900cf2e1086 WatchSource:0}: Error finding container 852e38394cd620480ee4f36fc7db4d4e260c01e31bd665e9837da900cf2e1086: Status 404 returned error can't find the container with id 852e38394cd620480ee4f36fc7db4d4e260c01e31bd665e9837da900cf2e1086 Nov 01 12:45:03.794902 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.794870 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerStarted","Data":"938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57"} Nov 01 12:45:03.796824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.796799 4484 generic.go:356] "Generic (PLEG): container finished" podID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerID="5b6bad30b8e8b25b1026f77361e937cb2bd47177bdd6b9c9dac60dda25867186" exitCode=60 Nov 01 12:45:03.796928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.796857 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" event={"ID":"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08","Type":"ContainerDied","Data":"5b6bad30b8e8b25b1026f77361e937cb2bd47177bdd6b9c9dac60dda25867186"} Nov 01 12:45:03.796928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.796906 4484 scope.go:122] "RemoveContainer" containerID="a6277b45be5e75002095c50844137520ab995808d615f7bbc6c694cf0221b290" Nov 01 12:45:03.797195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.797173 4484 scope.go:122] "RemoveContainer" containerID="5b6bad30b8e8b25b1026f77361e937cb2bd47177bdd6b9c9dac60dda25867186" Nov 01 12:45:03.797620 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:03.797594 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt_odh-ai-gateway-infra(56a53cee-aeba-4dbb-be9c-4e4dd7aafc08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" Nov 01 12:45:03.799008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.798988 4484 generic.go:356] "Generic (PLEG): container finished" podID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerID="1d2810518d5ae1a30e05f791a43a8b9c2cf3045981666de5261eb84711474313" exitCode=60 Nov 01 12:45:03.799090 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.799050 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" event={"ID":"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d","Type":"ContainerDied","Data":"1d2810518d5ae1a30e05f791a43a8b9c2cf3045981666de5261eb84711474313"} Nov 01 12:45:03.799339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.799295 4484 scope.go:122] "RemoveContainer" containerID="1d2810518d5ae1a30e05f791a43a8b9c2cf3045981666de5261eb84711474313" Nov 01 12:45:03.799616 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:03.799596 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c_odh-ai-gateway-infra(ee0a352f-a9ff-413e-be1b-fb3a6001fc8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" Nov 01 12:45:03.800729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.800709 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" event={"ID":"6cf8256d-ade3-4ce8-be64-404de4d0fc46","Type":"ContainerStarted","Data":"1a7fdfab7ba6dfc22f0545e3a68ba95f277035aef4419389c2252d7f4096e1f5"} Nov 01 12:45:03.800892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.800754 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" event={"ID":"6cf8256d-ade3-4ce8-be64-404de4d0fc46","Type":"ContainerStarted","Data":"852e38394cd620480ee4f36fc7db4d4e260c01e31bd665e9837da900cf2e1086"} Nov 01 12:45:03.863868 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:03.863847 4484 scope.go:122] "RemoveContainer" containerID="058afc9bce2195d3a79ee2944b3b020641634da955002ede06e3eebb16efcf02" Nov 01 12:45:04.130016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.129970 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:04.130996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.130957 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" podStartSLOduration=2.1309439 podStartE2EDuration="2.1309439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:04.127437538 +0000 UTC m=+1373.877380514" watchObservedRunningTime="2026-11-01 12:45:04.1309439 +0000 UTC m=+1373.880886878" Nov 01 12:45:04.335812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.335784 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225"] Nov 01 12:45:04.339911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.339892 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.389017 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.388980 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-78de6246-openshift-default-dockercfg-fxjjs\"" Nov 01 12:45:04.410397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.410360 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225"] Nov 01 12:45:04.446924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.446888 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:04.471225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471192 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.471564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471542 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ss4l\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.471713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471689 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.471783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471766 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.471841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471809 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.471911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471897 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.471999 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.471986 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.472048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.472029 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.472100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.472057 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580322 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ss4l\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580394 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580465 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580500 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580573 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580662 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580702 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580732 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.581373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.580877 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.588322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.582154 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.588322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.582747 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.588322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.582996 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.588322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.583802 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.588322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.584415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.608767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.607908 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.609961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.609929 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.642988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.642959 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ss4l\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.642988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.642973 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token\") pod \"e2e-ait-78de6246-openshift-default-b77d885-9w225\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.703656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.701087 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:04.708323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.706962 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:04.814128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.813882 4484 scope.go:122] "RemoveContainer" containerID="5b6bad30b8e8b25b1026f77361e937cb2bd47177bdd6b9c9dac60dda25867186" Nov 01 12:45:04.814682 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:04.814326 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt_odh-ai-gateway-infra(56a53cee-aeba-4dbb-be9c-4e4dd7aafc08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" Nov 01 12:45:04.816556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.816530 4484 scope.go:122] "RemoveContainer" containerID="1d2810518d5ae1a30e05f791a43a8b9c2cf3045981666de5261eb84711474313" Nov 01 12:45:04.816933 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:04.816909 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c_odh-ai-gateway-infra(ee0a352f-a9ff-413e-be1b-fb3a6001fc8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" Nov 01 12:45:04.818048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.818010 4484 generic.go:356] "Generic (PLEG): container finished" podID="6cf8256d-ade3-4ce8-be64-404de4d0fc46" containerID="1a7fdfab7ba6dfc22f0545e3a68ba95f277035aef4419389c2252d7f4096e1f5" exitCode=0 Nov 01 12:45:04.818161 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.818090 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" event={"ID":"6cf8256d-ade3-4ce8-be64-404de4d0fc46","Type":"ContainerDied","Data":"1a7fdfab7ba6dfc22f0545e3a68ba95f277035aef4419389c2252d7f4096e1f5"} Nov 01 12:45:04.820408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.820382 4484 generic.go:356] "Generic (PLEG): container finished" podID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerID="938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57" exitCode=60 Nov 01 12:45:04.821153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.820504 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerDied","Data":"938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57"} Nov 01 12:45:04.821153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.820560 4484 scope.go:122] "RemoveContainer" containerID="ad927d4054b6fd948547471224f6bf7267e0226a29e91b3af8fe48fad6138a04" Nov 01 12:45:04.821991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:04.820840 4484 scope.go:122] "RemoveContainer" containerID="938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57" Nov 01 12:45:04.821991 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:04.821941 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d_odh-ai-gateway-infra(f281d63b-66e2-49b1-bb8c-1c4d96b566d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" Nov 01 12:45:05.092748 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.092713 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm"] Nov 01 12:45:05.106747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.099155 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.148982 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.148952 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-83558e4e-openshift-default-dockercfg-6psfh\"" Nov 01 12:45:05.172030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.171996 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm"] Nov 01 12:45:05.190984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.190949 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191036 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhwv\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191124 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191217 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191354 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191418 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191498 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191483 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191524 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.191646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.191631 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.195238 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.195210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:05.205597 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.205562 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225"] Nov 01 12:45:05.213984 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:05.213958 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc69f0c7_b732_4628_94ba_bf43ee87cee6.slice/crio-74ef6c9871d5c2ee279522b3efac7251f2695cd2d3da2662080a8ef823911d66 WatchSource:0}: Error finding container 74ef6c9871d5c2ee279522b3efac7251f2695cd2d3da2662080a8ef823911d66: Status 404 returned error can't find the container with id 74ef6c9871d5c2ee279522b3efac7251f2695cd2d3da2662080a8ef823911d66 Nov 01 12:45:05.216312 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.216262 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:05.216415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.216358 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:05.216415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.216406 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293381 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293472 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293541 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293675 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293722 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293790 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhwv\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.293918 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.296326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.296117 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.304088 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.303725 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.323323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.312046 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.323956 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.323860 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.324265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.324225 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.335721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.335656 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.338590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.338529 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.396318 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.391655 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhwv\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.400967 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.400908 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token\") pod \"e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.419128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.419093 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:05.613357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.613181 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:05.832619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.832576 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" event={"ID":"bc69f0c7-b732-4628-94ba-bf43ee87cee6","Type":"ContainerStarted","Data":"61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d"} Nov 01 12:45:05.833098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.832637 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" event={"ID":"bc69f0c7-b732-4628-94ba-bf43ee87cee6","Type":"ContainerStarted","Data":"74ef6c9871d5c2ee279522b3efac7251f2695cd2d3da2662080a8ef823911d66"} Nov 01 12:45:05.836158 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.835739 4484 scope.go:122] "RemoveContainer" containerID="938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57" Nov 01 12:45:05.836158 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:05.836129 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d_odh-ai-gateway-infra(f281d63b-66e2-49b1-bb8c-1c4d96b566d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" Nov 01 12:45:05.838809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.838584 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm"] Nov 01 12:45:05.843398 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:05.843369 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b768de_eab0_41ed_b2ee_4a7baf0122de.slice/crio-3d2b9c9fc736f3078a866fa745a1b36167cd6cc5579dc1dc427fd74d9b49358d WatchSource:0}: Error finding container 3d2b9c9fc736f3078a866fa745a1b36167cd6cc5579dc1dc427fd74d9b49358d: Status 404 returned error can't find the container with id 3d2b9c9fc736f3078a866fa745a1b36167cd6cc5579dc1dc427fd74d9b49358d Nov 01 12:45:05.847456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.847414 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:05.847575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.847505 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:05.847575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.847558 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:05.954241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:05.954180 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podStartSLOduration=1.954162249 podStartE2EDuration="1.954162249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:05.953986127 +0000 UTC m=+1375.703929110" watchObservedRunningTime="2026-11-01 12:45:05.954162249 +0000 UTC m=+1375.704105228" Nov 01 12:45:06.029024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.028992 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:06.120371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.113863 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca\") pod \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " Nov 01 12:45:06.120371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.114255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca\") pod \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " Nov 01 12:45:06.120371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.114316 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq\" (UniqueName: \"kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq\") pod \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\" (UID: \"6cf8256d-ade3-4ce8-be64-404de4d0fc46\") " Nov 01 12:45:06.125739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.125677 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca" pod "6cf8256d-ade3-4ce8-be64-404de4d0fc46" (UID: "6cf8256d-ade3-4ce8-be64-404de4d0fc46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:45:06.136447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.126913 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca" pod "6cf8256d-ade3-4ce8-be64-404de4d0fc46" (UID: "6cf8256d-ade3-4ce8-be64-404de4d0fc46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:45:06.137080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.137047 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq" pod "6cf8256d-ade3-4ce8-be64-404de4d0fc46" (UID: "6cf8256d-ade3-4ce8-be64-404de4d0fc46"). InnerVolumeSpecName "kube-api-access-smdxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:45:06.219331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.217144 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:06.219331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.217179 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdxq\" (UniqueName: \"kubernetes.io/projected/6cf8256d-ade3-4ce8-be64-404de4d0fc46-kube-api-access-smdxq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:06.219331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.217196 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf8256d-ade3-4ce8-be64-404de4d0fc46-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:06.702105 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.702015 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:06.707540 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.707502 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:06.845543 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.845508 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" event={"ID":"59b768de-eab0-41ed-b2ee-4a7baf0122de","Type":"ContainerStarted","Data":"95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82"} Nov 01 12:45:06.845936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.845600 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" event={"ID":"59b768de-eab0-41ed-b2ee-4a7baf0122de","Type":"ContainerStarted","Data":"3d2b9c9fc736f3078a866fa745a1b36167cd6cc5579dc1dc427fd74d9b49358d"} Nov 01 12:45:06.846853 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.846834 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" Nov 01 12:45:06.846962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.846868 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h" event={"ID":"6cf8256d-ade3-4ce8-be64-404de4d0fc46","Type":"ContainerDied","Data":"852e38394cd620480ee4f36fc7db4d4e260c01e31bd665e9837da900cf2e1086"} Nov 01 12:45:06.846962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.846911 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852e38394cd620480ee4f36fc7db4d4e260c01e31bd665e9837da900cf2e1086" Nov 01 12:45:06.847335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.847281 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:06.848311 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.848279 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:45:06.959166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:06.958943 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podStartSLOduration=1.958926787 podStartE2EDuration="1.958926787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:06.957860102 +0000 UTC m=+1376.707803085" watchObservedRunningTime="2026-11-01 12:45:06.958926787 +0000 UTC m=+1376.708869766" Nov 01 12:45:07.420083 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:07.420050 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:07.425779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:07.425750 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:07.430168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:07.430141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:07.853558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:07.853526 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:07.854270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:07.854252 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:45:08.102437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:08.102400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:08.290408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:08.290378 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:08.460361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:08.460333 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:08.613403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:08.613284 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:09.148274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:09.148218 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:09.400381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:09.392190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:09.556323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:09.551039 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:09.755093 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:09.755051 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:10.388461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:10.388432 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:10.945949 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:10.945919 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:11.159437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:11.159380 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:13.228242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.228210 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv"] Nov 01 12:45:13.228877 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.228862 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6cf8256d-ade3-4ce8-be64-404de4d0fc46" containerName="revoke-keys" Nov 01 12:45:13.228924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.228879 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf8256d-ade3-4ce8-be64-404de4d0fc46" containerName="revoke-keys" Nov 01 12:45:13.229037 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.229026 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf8256d-ade3-4ce8-be64-404de4d0fc46" containerName="revoke-keys" Nov 01 12:45:13.234488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.234468 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.264395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.264358 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.264576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.264413 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt75p\" (UniqueName: \"kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.264701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.264684 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.301000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.300968 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:45:13.334962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.334929 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv"] Nov 01 12:45:13.365766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.365735 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.365960 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.365779 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt75p\" (UniqueName: \"kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.365960 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.365945 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.366631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.366607 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.368286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.368259 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.426402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.426365 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt75p\" (UniqueName: \"kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p\") pod \"maas-api-revoke-sub-befbc351-978zv\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.548200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.548120 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:13.936744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:13.936666 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv"] Nov 01 12:45:13.942235 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:13.942212 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7658bfe_9033_4d52_960d_6086b78082a2.slice/crio-d5883dce98536ae0dfc7e5d332e1ce8ec24a201c7788fae1bac05cf130e0f731 WatchSource:0}: Error finding container d5883dce98536ae0dfc7e5d332e1ce8ec24a201c7788fae1bac05cf130e0f731: Status 404 returned error can't find the container with id d5883dce98536ae0dfc7e5d332e1ce8ec24a201c7788fae1bac05cf130e0f731 Nov 01 12:45:14.673679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:14.673610 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:14.874695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:14.874663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:14.902410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:14.902364 4484 generic.go:356] "Generic (PLEG): container finished" podID="e7658bfe-9033-4d52-960d-6086b78082a2" containerID="2f5a6a2c538138ca8e483f1d9db232df8892b3ac676f0bcae620f2d2229048ad" exitCode=0 Nov 01 12:45:14.902629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:14.902496 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" event={"ID":"e7658bfe-9033-4d52-960d-6086b78082a2","Type":"ContainerDied","Data":"2f5a6a2c538138ca8e483f1d9db232df8892b3ac676f0bcae620f2d2229048ad"} Nov 01 12:45:14.902629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:14.902541 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" event={"ID":"e7658bfe-9033-4d52-960d-6086b78082a2","Type":"ContainerStarted","Data":"d5883dce98536ae0dfc7e5d332e1ce8ec24a201c7788fae1bac05cf130e0f731"} Nov 01 12:45:15.145370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:15.145338 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:15.362359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:15.362328 4484 scope.go:122] "RemoveContainer" containerID="938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57" Nov 01 12:45:15.911533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:15.911446 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerStarted","Data":"92e57a5a817df2da54678c85bdc46a40d89897392f090b78a2992e5343c3a0f7"} Nov 01 12:45:16.061333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.061311 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:16.104282 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.104255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p\" (UniqueName: \"kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p\") pod \"e7658bfe-9033-4d52-960d-6086b78082a2\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " Nov 01 12:45:16.104460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.104448 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca\") pod \"e7658bfe-9033-4d52-960d-6086b78082a2\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " Nov 01 12:45:16.104560 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.104546 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca\") pod \"e7658bfe-9033-4d52-960d-6086b78082a2\" (UID: \"e7658bfe-9033-4d52-960d-6086b78082a2\") " Nov 01 12:45:16.104852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.104827 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca" pod "e7658bfe-9033-4d52-960d-6086b78082a2" (UID: "e7658bfe-9033-4d52-960d-6086b78082a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:45:16.105012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.104997 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:16.106223 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.106203 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p" pod "e7658bfe-9033-4d52-960d-6086b78082a2" (UID: "e7658bfe-9033-4d52-960d-6086b78082a2"). InnerVolumeSpecName "kube-api-access-bt75p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:45:16.106483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.106466 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca" pod "e7658bfe-9033-4d52-960d-6086b78082a2" (UID: "e7658bfe-9033-4d52-960d-6086b78082a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:45:16.206515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.206442 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7658bfe-9033-4d52-960d-6086b78082a2-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:16.206515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.206480 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt75p\" (UniqueName: \"kubernetes.io/projected/e7658bfe-9033-4d52-960d-6086b78082a2-kube-api-access-bt75p\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:16.416759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.416731 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:16.650618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.650578 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:16.942374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.932963 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv"] Nov 01 12:45:16.942374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.934097 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e7658bfe-9033-4d52-960d-6086b78082a2" containerName="revoke-keys" Nov 01 12:45:16.942374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.934124 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7658bfe-9033-4d52-960d-6086b78082a2" containerName="revoke-keys" Nov 01 12:45:16.942374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.934400 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7658bfe-9033-4d52-960d-6086b78082a2" containerName="revoke-keys" Nov 01 12:45:16.942374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.938642 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:16.960334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.954692 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:16.960334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.955714 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wf4\" (UniqueName: \"kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4\") pod \"payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:16.960334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:16.955941 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:17.172333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.146889 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wf4\" (UniqueName: \"kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4\") pod \"payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:17.220377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.173790 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-78de6246\"" Nov 01 12:45:17.220377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.189255 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv"] Nov 01 12:45:17.249125 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.249087 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:17.249614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.219889 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:17.317578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.312553 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-78de6246-dockercfg-64vc8\"" Nov 01 12:45:17.319430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.319406 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l"] Nov 01 12:45:17.356218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.356183 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wf4\" (UniqueName: \"kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4\") pod \"payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:17.377722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.369080 4484 generic.go:356] "Generic (PLEG): container finished" podID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerID="92e57a5a817df2da54678c85bdc46a40d89897392f090b78a2992e5343c3a0f7" exitCode=60 Nov 01 12:45:17.377722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.371496 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" Nov 01 12:45:17.400426 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.400399 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:17.402955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.402931 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l"] Nov 01 12:45:17.403118 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.403107 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx"] Nov 01 12:45:17.404800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.404608 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.412576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.412450 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerDied","Data":"92e57a5a817df2da54678c85bdc46a40d89897392f090b78a2992e5343c3a0f7"} Nov 01 12:45:17.412576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.412541 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.412871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.412635 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d"] Nov 01 12:45:17.412871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.412718 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv" event={"ID":"e7658bfe-9033-4d52-960d-6086b78082a2","Type":"ContainerDied","Data":"d5883dce98536ae0dfc7e5d332e1ce8ec24a201c7788fae1bac05cf130e0f731"} Nov 01 12:45:17.412871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.412740 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5883dce98536ae0dfc7e5d332e1ce8ec24a201c7788fae1bac05cf130e0f731" Nov 01 12:45:17.413030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.412984 4484 scope.go:122] "RemoveContainer" containerID="938a456cab72e386ca3f4987a952f1b9481baa4689b70d0a76179db67b6b4b57" Nov 01 12:45:17.413845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.413822 4484 scope.go:122] "RemoveContainer" containerID="5b6bad30b8e8b25b1026f77361e937cb2bd47177bdd6b9c9dac60dda25867186" Nov 01 12:45:17.414868 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.414854 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx"] Nov 01 12:45:17.415200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.415165 4484 scope.go:122] "RemoveContainer" containerID="92e57a5a817df2da54678c85bdc46a40d89897392f090b78a2992e5343c3a0f7" Nov 01 12:45:17.415492 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:17.415472 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d_odh-ai-gateway-infra(f281d63b-66e2-49b1-bb8c-1c4d96b566d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" Nov 01 12:45:17.456421 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.456384 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.456536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.456503 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.456812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.456789 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75n4n\" (UniqueName: \"kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n\") pod \"payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.457032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457016 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.457166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457151 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.457372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457353 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.457484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457469 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.457664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457640 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.457734 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457694 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume\") pod \"payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.457804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457730 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwxr\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.457906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.457880 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559179 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559143 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75n4n\" (UniqueName: \"kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n\") pod \"payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.559368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559284 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559378 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559432 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559457 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559547 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559593 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume\") pod \"payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.559688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559623 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwxr\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559677 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559778 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.559847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.559811 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.560084 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.560056 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.560172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.560087 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.560172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.560141 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.560374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.560351 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.560481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.560452 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.560840 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.560816 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume\") pod \"payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.562459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.562427 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.562549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.562520 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.614933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.614901 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75n4n\" (UniqueName: \"kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n\") pod \"payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.615125 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.615105 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.615179 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.615104 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwxr\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:17.777173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.777145 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:17.792677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.792653 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv"] Nov 01 12:45:17.797801 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:17.797774 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ee3cfd_3cf7_4ad2_b766_0b9e2bdae1bf.slice/crio-7f58c815bfec71a3369974c16790fa825c6e651a4e9d8ba58e4becbe33b1a589 WatchSource:0}: Error finding container 7f58c815bfec71a3369974c16790fa825c6e651a4e9d8ba58e4becbe33b1a589: Status 404 returned error can't find the container with id 7f58c815bfec71a3369974c16790fa825c6e651a4e9d8ba58e4becbe33b1a589 Nov 01 12:45:17.798684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:17.798657 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:18.164587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.164514 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l"] Nov 01 12:45:18.169974 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:18.169943 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fda319_5fdc_404e_b79a_cafa8d1a0e65.slice/crio-ff79431f9c9ac7f53c5fa134b8155c3a7055b9feb29c0389ed2ba9627043aa90 WatchSource:0}: Error finding container ff79431f9c9ac7f53c5fa134b8155c3a7055b9feb29c0389ed2ba9627043aa90: Status 404 returned error can't find the container with id ff79431f9c9ac7f53c5fa134b8155c3a7055b9feb29c0389ed2ba9627043aa90 Nov 01 12:45:18.285750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.285719 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx"] Nov 01 12:45:18.296312 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:18.296272 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc76e96d_8c09_4f99_a9bd_0f526f7016a3.slice/crio-dc842d9f606569433e85289d506d8bd696a3e247fe300913aa2d60c739c1a5cf WatchSource:0}: Error finding container dc842d9f606569433e85289d506d8bd696a3e247fe300913aa2d60c739c1a5cf: Status 404 returned error can't find the container with id dc842d9f606569433e85289d506d8bd696a3e247fe300913aa2d60c739c1a5cf Nov 01 12:45:18.298825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.298790 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:18.298889 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.298859 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:18.298930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.298887 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:18.362849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.362827 4484 scope.go:122] "RemoveContainer" containerID="1d2810518d5ae1a30e05f791a43a8b9c2cf3045981666de5261eb84711474313" Nov 01 12:45:18.381597 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.381562 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" event={"ID":"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf","Type":"ContainerStarted","Data":"3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea"} Nov 01 12:45:18.381698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.381627 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" event={"ID":"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf","Type":"ContainerStarted","Data":"7f58c815bfec71a3369974c16790fa825c6e651a4e9d8ba58e4becbe33b1a589"} Nov 01 12:45:18.381920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.381895 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:18.383587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.383557 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" event={"ID":"90fda319-5fdc-404e-b79a-cafa8d1a0e65","Type":"ContainerStarted","Data":"5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8"} Nov 01 12:45:18.383703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.383611 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" event={"ID":"90fda319-5fdc-404e-b79a-cafa8d1a0e65","Type":"ContainerStarted","Data":"ff79431f9c9ac7f53c5fa134b8155c3a7055b9feb29c0389ed2ba9627043aa90"} Nov 01 12:45:18.383703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.383665 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:18.388666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.388641 4484 generic.go:356] "Generic (PLEG): container finished" podID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerID="4e745454f43108650d374bcd0b27ecf6e895a07c97f4a90cd7a9186ebfcf64ae" exitCode=60 Nov 01 12:45:18.388754 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.388719 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" event={"ID":"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08","Type":"ContainerDied","Data":"4e745454f43108650d374bcd0b27ecf6e895a07c97f4a90cd7a9186ebfcf64ae"} Nov 01 12:45:18.388828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.388784 4484 scope.go:122] "RemoveContainer" containerID="5b6bad30b8e8b25b1026f77361e937cb2bd47177bdd6b9c9dac60dda25867186" Nov 01 12:45:18.389171 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.389155 4484 scope.go:122] "RemoveContainer" containerID="4e745454f43108650d374bcd0b27ecf6e895a07c97f4a90cd7a9186ebfcf64ae" Nov 01 12:45:18.389485 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:18.389460 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt_odh-ai-gateway-infra(56a53cee-aeba-4dbb-be9c-4e4dd7aafc08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" Nov 01 12:45:18.391465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.391426 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" event={"ID":"bc76e96d-8c09-4f99-a9bd-0f526f7016a3","Type":"ContainerStarted","Data":"a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b"} Nov 01 12:45:18.391641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.391467 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" event={"ID":"bc76e96d-8c09-4f99-a9bd-0f526f7016a3","Type":"ContainerStarted","Data":"dc842d9f606569433e85289d506d8bd696a3e247fe300913aa2d60c739c1a5cf"} Nov 01 12:45:18.498432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.498373 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" podStartSLOduration=2.498354737 podStartE2EDuration="2.498354737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:18.494851346 +0000 UTC m=+1388.244794317" watchObservedRunningTime="2026-11-01 12:45:18.498354737 +0000 UTC m=+1388.248297715" Nov 01 12:45:18.564091 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.564060 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:18.633008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.612640 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podStartSLOduration=2.61262304 podStartE2EDuration="2.61262304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:18.609329276 +0000 UTC m=+1388.359272251" watchObservedRunningTime="2026-11-01 12:45:18.61262304 +0000 UTC m=+1388.362566034" Nov 01 12:45:18.681324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.677565 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca\") pod \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " Nov 01 12:45:18.681324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.677634 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv\" (UniqueName: \"kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv\") pod \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\" (UID: \"f281d63b-66e2-49b1-bb8c-1c4d96b566d5\") " Nov 01 12:45:18.681324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.679550 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca" pod "f281d63b-66e2-49b1-bb8c-1c4d96b566d5" (UID: "f281d63b-66e2-49b1-bb8c-1c4d96b566d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:45:18.685483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.683444 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv" pod "f281d63b-66e2-49b1-bb8c-1c4d96b566d5" (UID: "f281d63b-66e2-49b1-bb8c-1c4d96b566d5"). InnerVolumeSpecName "kube-api-access-w6mfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:45:18.779994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.779958 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:18.779994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.779988 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6mfv\" (UniqueName: \"kubernetes.io/projected/f281d63b-66e2-49b1-bb8c-1c4d96b566d5-kube-api-access-w6mfv\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:18.798932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.798901 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:18.865677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:18.865620 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" podStartSLOduration=2.865601345 podStartE2EDuration="2.865601345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:18.860164722 +0000 UTC m=+1388.610107702" watchObservedRunningTime="2026-11-01 12:45:18.865601345 +0000 UTC m=+1388.615544324" Nov 01 12:45:19.398866 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.398843 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" Nov 01 12:45:19.399230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.398866 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d" event={"ID":"f281d63b-66e2-49b1-bb8c-1c4d96b566d5","Type":"ContainerDied","Data":"454810bc6e2b0c03738353ee4f31e9d1b07798876484df38b9283f9b3c3aafac"} Nov 01 12:45:19.399230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.398927 4484 scope.go:122] "RemoveContainer" containerID="92e57a5a817df2da54678c85bdc46a40d89897392f090b78a2992e5343c3a0f7" Nov 01 12:45:19.403421 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.403398 4484 generic.go:356] "Generic (PLEG): container finished" podID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerID="b79f6a9ca5cc823a36a4853904c24436b73f340529d18703abd8f51f2df8bd4e" exitCode=60 Nov 01 12:45:19.403557 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.403497 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" event={"ID":"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d","Type":"ContainerDied","Data":"b79f6a9ca5cc823a36a4853904c24436b73f340529d18703abd8f51f2df8bd4e"} Nov 01 12:45:19.404188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.404168 4484 scope.go:122] "RemoveContainer" containerID="b79f6a9ca5cc823a36a4853904c24436b73f340529d18703abd8f51f2df8bd4e" Nov 01 12:45:19.404739 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:19.404713 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c_odh-ai-gateway-infra(ee0a352f-a9ff-413e-be1b-fb3a6001fc8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" Nov 01 12:45:19.411068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.410864 4484 scope.go:122] "RemoveContainer" containerID="1d2810518d5ae1a30e05f791a43a8b9c2cf3045981666de5261eb84711474313" Nov 01 12:45:19.574368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.574332 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d"] Nov 01 12:45:19.582495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.582470 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c7206f-29892285-x755d"] Nov 01 12:45:19.771194 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.771156 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt"] Nov 01 12:45:19.802092 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.802055 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:19.802274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.802118 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:19.981108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:19.981083 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:20.001352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.001324 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca\") pod \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " Nov 01 12:45:20.001519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.001507 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx\" (UniqueName: \"kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx\") pod \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\" (UID: \"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08\") " Nov 01 12:45:20.001665 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.001646 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca" pod "56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" (UID: "56a53cee-aeba-4dbb-be9c-4e4dd7aafc08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:45:20.002002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.001987 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:20.003603 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.003579 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx" pod "56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" (UID: "56a53cee-aeba-4dbb-be9c-4e4dd7aafc08"). InnerVolumeSpecName "kube-api-access-9rmpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:45:20.103431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.103362 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rmpx\" (UniqueName: \"kubernetes.io/projected/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08-kube-api-access-9rmpx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:20.382867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.382789 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" path="/var/lib/kubelet/pods/f281d63b-66e2-49b1-bb8c-1c4d96b566d5/volumes" Nov 01 12:45:20.417857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.417827 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" Nov 01 12:45:20.417857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.417850 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt" event={"ID":"56a53cee-aeba-4dbb-be9c-4e4dd7aafc08","Type":"ContainerDied","Data":"3194de6c8537583f4acd60dbc53fe2e8dc0a0f718d8b0215788c4ec2aeb71c35"} Nov 01 12:45:20.418341 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.417907 4484 scope.go:122] "RemoveContainer" containerID="4e745454f43108650d374bcd0b27ecf6e895a07c97f4a90cd7a9186ebfcf64ae" Nov 01 12:45:20.584598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.584564 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt"] Nov 01 12:45:20.588507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.588478 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892285-vmgjt"] Nov 01 12:45:20.768364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.768329 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c"] Nov 01 12:45:20.799641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.799602 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:20.799812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.799667 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:20.982568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:20.982529 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:21.019807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.019727 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j\" (UniqueName: \"kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j\") pod \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " Nov 01 12:45:21.019958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.019833 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca\") pod \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\" (UID: \"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d\") " Nov 01 12:45:21.020221 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.020188 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca" pod "ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" (UID: "ee0a352f-a9ff-413e-be1b-fb3a6001fc8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:45:21.020477 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.020456 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:21.021993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.021969 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j" pod "ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" (UID: "ee0a352f-a9ff-413e-be1b-fb3a6001fc8d"). InnerVolumeSpecName "kube-api-access-65f9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:45:21.122075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.122031 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65f9j\" (UniqueName: \"kubernetes.io/projected/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d-kube-api-access-65f9j\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:45:21.426548 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.426466 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" Nov 01 12:45:21.426548 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.426484 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c" event={"ID":"ee0a352f-a9ff-413e-be1b-fb3a6001fc8d","Type":"ContainerDied","Data":"cf1a6cf74f8a6aca569767a0427ae93f730e09daaf9345d4997866ce33abfe6f"} Nov 01 12:45:21.427033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.426549 4484 scope.go:122] "RemoveContainer" containerID="b79f6a9ca5cc823a36a4853904c24436b73f340529d18703abd8f51f2df8bd4e" Nov 01 12:45:21.587670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.587637 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c"] Nov 01 12:45:21.592752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.592722 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892285-2ct2c"] Nov 01 12:45:21.796862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.796827 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:21.799949 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.799924 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:21.800055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.799986 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:21.995754 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:21.995709 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:22.326333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.316864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:22.610268 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.600446 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" path="/var/lib/kubelet/pods/56a53cee-aeba-4dbb-be9c-4e4dd7aafc08/volumes" Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.700850 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" path="/var/lib/kubelet/pods/ee0a352f-a9ff-413e-be1b-fb3a6001fc8d/volumes" Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.701358 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr"] Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.702229 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.702245 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.702262 4484 cpu_manager.go:414] "containerMap: removing container" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.702270 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.704319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.702289 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727623 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727666 4484 cpu_manager.go:414] "containerMap: removing container" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727680 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727698 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727706 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727778 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727786 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727819 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.727826 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728044 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728062 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728073 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728084 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728096 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728107 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728117 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f281d63b-66e2-49b1-bb8c-1c4d96b566d5" containerName="cleanup" Nov 01 12:45:22.729935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.728127 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.860246 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.849502 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:22.860246 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.849571 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:22.895494 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.881390 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr"] Nov 01 12:45:22.895494 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.881448 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h"] Nov 01 12:45:22.925987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.912991 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:22.925987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.914107 4484 cpu_manager.go:414] "containerMap: removing container" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.925987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.914127 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a53cee-aeba-4dbb-be9c-4e4dd7aafc08" containerName="cleanup" Nov 01 12:45:22.925987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.914141 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.925987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.914192 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:22.925987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:22.914428 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0a352f-a9ff-413e-be1b-fb3a6001fc8d" containerName="cleanup" Nov 01 12:45:23.008879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.004271 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhhg\" (UniqueName: \"kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg\") pod \"payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.008879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.004462 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.008879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.008215 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.024738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.024712 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h"] Nov 01 12:45:23.027409 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.027385 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-83558e4e-dockercfg-2lqcz\"" Nov 01 12:45:23.027530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.027448 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-83558e4e\"" Nov 01 12:45:23.105570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.105539 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.105778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.105604 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6t2\" (UniqueName: \"kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2\") pod \"payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.105778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.105625 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume\") pod \"payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.105905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.105884 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhhg\" (UniqueName: \"kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg\") pod \"payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.106745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.106725 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.170861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.170796 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhhg\" (UniqueName: \"kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg\") pod \"payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.207846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.207820 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6t2\" (UniqueName: \"kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2\") pod \"payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.207977 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.207854 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume\") pod \"payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.208662 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.208641 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume\") pod \"payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.267837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.267806 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6t2\" (UniqueName: \"kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2\") pod \"payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.290598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.290572 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:23.331113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.331089 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:23.536611 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.536539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:23.710701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.710662 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr"] Nov 01 12:45:23.711533 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:23.711489 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6623ea9d_984a_4597_8fba_d27479f83fc7.slice/crio-781a8e8fd710c95d4b2e926244f221f27c5fdae64c44a62c32d4ae05a871f03e WatchSource:0}: Error finding container 781a8e8fd710c95d4b2e926244f221f27c5fdae64c44a62c32d4ae05a871f03e: Status 404 returned error can't find the container with id 781a8e8fd710c95d4b2e926244f221f27c5fdae64c44a62c32d4ae05a871f03e Nov 01 12:45:23.723893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.723867 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" event={"ID":"6623ea9d-984a-4597-8fba-d27479f83fc7","Type":"ContainerStarted","Data":"781a8e8fd710c95d4b2e926244f221f27c5fdae64c44a62c32d4ae05a871f03e"} Nov 01 12:45:23.799214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.799095 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:23.799214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.799140 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:23.809458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:23.809433 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h"] Nov 01 12:45:23.814604 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:23.814579 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49178b94_1035_4efd_a9c5_fc616bd00e77.slice/crio-f42a858f8349e578d02890a4a56caa46779f94bd003f97b29f060705c1161e21 WatchSource:0}: Error finding container f42a858f8349e578d02890a4a56caa46779f94bd003f97b29f060705c1161e21: Status 404 returned error can't find the container with id f42a858f8349e578d02890a4a56caa46779f94bd003f97b29f060705c1161e21 Nov 01 12:45:24.733915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.733877 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" event={"ID":"6623ea9d-984a-4597-8fba-d27479f83fc7","Type":"ContainerStarted","Data":"825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887"} Nov 01 12:45:24.734342 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.733950 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:24.735750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.735713 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" event={"ID":"49178b94-1035-4efd-a9c5-fc616bd00e77","Type":"ContainerStarted","Data":"2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee"} Nov 01 12:45:24.735867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.735793 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" event={"ID":"49178b94-1035-4efd-a9c5-fc616bd00e77","Type":"ContainerStarted","Data":"f42a858f8349e578d02890a4a56caa46779f94bd003f97b29f060705c1161e21"} Nov 01 12:45:24.735867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.735818 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:24.799255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.799224 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:24.799454 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.799279 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:24.853637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.853567 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" podStartSLOduration=2.8535483040000003 podStartE2EDuration="2.853548304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:24.846711271 +0000 UTC m=+1394.596654249" watchObservedRunningTime="2026-11-01 12:45:24.853548304 +0000 UTC m=+1394.603491282" Nov 01 12:45:24.921956 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.921927 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:24.968507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:24.968451 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" podStartSLOduration=2.968439196 podStartE2EDuration="2.968439196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:24.966202167 +0000 UTC m=+1394.716145146" watchObservedRunningTime="2026-11-01 12:45:24.968439196 +0000 UTC m=+1394.718382193" Nov 01 12:45:25.146659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.146576 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:25.381158 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.378862 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7"] Nov 01 12:45:25.422369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.422175 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.435733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.430918 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7"] Nov 01 12:45:25.464776 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.464741 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.464945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.464842 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98sc\" (UniqueName: \"kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc\") pod \"payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.477344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.476642 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n"] Nov 01 12:45:25.485887 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.485193 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.485887 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.485194 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b2ccbb55\"" Nov 01 12:45:25.523326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.517544 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b2ccbb55-dockercfg-t7lkh\"" Nov 01 12:45:25.568327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.567277 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98sc\" (UniqueName: \"kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc\") pod \"payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.568327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.567475 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpr26\" (UniqueName: \"kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26\") pod \"payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.568327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.567616 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume\") pod \"payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.568327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.567675 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.574027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.568662 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.710374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.628112 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n"] Nov 01 12:45:25.720753 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.720060 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpr26\" (UniqueName: \"kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26\") pod \"payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.766482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.730245 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume\") pod \"payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.766482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.728754 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume\") pod \"payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.808894 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.808793 4484 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 12:45:25.812405 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.808866 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 12:45:25.821818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.821780 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98sc\" (UniqueName: \"kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc\") pod \"payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:25.839984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.838406 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpr26\" (UniqueName: \"kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26\") pod \"payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:25.852356 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:25.852327 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:26.053523 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.053493 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:26.292653 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.292395 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n"] Nov 01 12:45:26.294893 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:26.294865 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1b75ef_a924_4f95_aaa9_355bc8a40cc0.slice/crio-d86672c3799c81d4f93d3b1483122bc64c3a2fc631d1eed945fbb4d15b3baef2 WatchSource:0}: Error finding container d86672c3799c81d4f93d3b1483122bc64c3a2fc631d1eed945fbb4d15b3baef2: Status 404 returned error can't find the container with id d86672c3799c81d4f93d3b1483122bc64c3a2fc631d1eed945fbb4d15b3baef2 Nov 01 12:45:26.454162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.454124 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7"] Nov 01 12:45:26.458623 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:26.458596 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1fcba7_129d_4fdb_80f0_d152e7c72c01.slice/crio-351b9316134b7b94ed4d628396aeeccb25de4e46209460558ba48442406963e2 WatchSource:0}: Error finding container 351b9316134b7b94ed4d628396aeeccb25de4e46209460558ba48442406963e2: Status 404 returned error can't find the container with id 351b9316134b7b94ed4d628396aeeccb25de4e46209460558ba48442406963e2 Nov 01 12:45:26.628848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.628748 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:26.785020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.784979 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" event={"ID":"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0","Type":"ContainerStarted","Data":"a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90"} Nov 01 12:45:26.785509 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.785048 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:26.785509 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.785074 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" event={"ID":"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0","Type":"ContainerStarted","Data":"d86672c3799c81d4f93d3b1483122bc64c3a2fc631d1eed945fbb4d15b3baef2"} Nov 01 12:45:26.786737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.786711 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" event={"ID":"9e1fcba7-129d-4fdb-80f0-d152e7c72c01","Type":"ContainerStarted","Data":"5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50"} Nov 01 12:45:26.786859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.786754 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" event={"ID":"9e1fcba7-129d-4fdb-80f0-d152e7c72c01","Type":"ContainerStarted","Data":"351b9316134b7b94ed4d628396aeeccb25de4e46209460558ba48442406963e2"} Nov 01 12:45:26.786859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.786773 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:26.804067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.804047 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:26.804339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.804321 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:26.804943 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.804919 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:45:26.852542 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:26.852516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:27.044602 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:27.030937 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" podStartSLOduration=2.030916411 podStartE2EDuration="2.030916411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:26.904264044 +0000 UTC m=+1396.654207023" watchObservedRunningTime="2026-11-01 12:45:27.030916411 +0000 UTC m=+1396.780859390" Nov 01 12:45:27.204322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:27.196225 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" podStartSLOduration=2.196207147 podStartE2EDuration="2.196207147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:27.070073023 +0000 UTC m=+1396.820016050" watchObservedRunningTime="2026-11-01 12:45:27.196207147 +0000 UTC m=+1396.946150125" Nov 01 12:45:27.922098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:27.922064 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:28.200594 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:28.200516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:29.406928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:29.406893 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:45:29.407362 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:29.406982 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:45:29.487777 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:29.487746 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:30.658705 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:45:30.658660 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:45:31.212324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:31.211394 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 12:45:31.264452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:31.264397 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:31.280417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:31.280373 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh" podUID="b95d7585-b83b-4e4a-bfac-cf3c16c721d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:31.383496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:31.383456 4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:31.841012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:31.837641 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:31.876185 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:31.875404 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:32.059249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.059178 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:32.214563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.214439 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:32.216166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.216133 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:32.216535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.216517 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:45:32.867841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.843471 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:32.867841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.844045 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:45:32.867841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.867422 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:32.958141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.892386 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:45:32.958141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:32.892704 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:45:33.060550 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.060494 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:33.073648 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.071742 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:45:33.483325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.437499 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 12:45:33.483325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.442834 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:45:33.635325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.616998 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:33.881381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.837433 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:33.936849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.908734 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 01 12:45:33.936849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:33.909712 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 12:45:34.177383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.173522 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 12:45:34.528465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.528394 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:34.530654 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.530605 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:34.660356 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.654208 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 12:45:34.665317 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.661950 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 12:45:34.719281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.719227 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:34.910380 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.861208 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:34.913551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:34.872518 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 12:45:35.000604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.000562 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:35.065368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.065325 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:45:35.198696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.198611 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:45:35.198959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.198941 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 12:45:35.199100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.199087 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 12:45:35.199219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.199206 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 12:45:35.199401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.199384 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:45:35.199555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.199542 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:45:35.260613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.260585 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 12:45:35.779910 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.779870 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:45:35.780109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:35.779951 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:45:37.796770 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:37.796742 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:45:37.797216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:37.796808 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:45:37.988448 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:37.988414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:43.452317 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:43.452276 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:46.277784 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:46.277748 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:47.138070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.138037 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j"] Nov 01 12:45:47.143771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.143743 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.184504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.184480 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs"] Nov 01 12:45:47.188947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.188930 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.202596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.202577 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bc8bt\"" Nov 01 12:45:47.233578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233549 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccchn\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.233674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233587 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwwj\" (UniqueName: \"kubernetes.io/projected/1f602e80-772b-439c-a04c-526955a37d83-kube-api-access-qmwwj\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.233674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233622 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.233780 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233743 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.233869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233848 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.233938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233913 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.233993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233954 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.234047 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.233998 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.234047 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234028 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.234154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234079 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.234154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234121 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f602e80-772b-439c-a04c-526955a37d83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.234154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234148 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.234338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234176 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.234338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.234338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234281 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.234495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234346 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f602e80-772b-439c-a04c-526955a37d83-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.234601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234581 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f602e80-772b-439c-a04c-526955a37d83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.234655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.234631 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.238913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.238877 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j"] Nov 01 12:45:47.290590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.290565 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs"] Nov 01 12:45:47.336217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336183 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccchn\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwwj\" (UniqueName: \"kubernetes.io/projected/1f602e80-772b-439c-a04c-526955a37d83-kube-api-access-qmwwj\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336257 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336286 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336363 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336389 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336423 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336465 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336493 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336550 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336593 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f602e80-772b-439c-a04c-526955a37d83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336624 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336648 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336718 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336759 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.336907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336803 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f602e80-772b-439c-a04c-526955a37d83-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336832 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336875 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f602e80-772b-439c-a04c-526955a37d83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.336907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336898 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.337159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336934 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.337159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337041 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.337319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337266 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.337319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337286 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.337452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.336760 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.337584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337561 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.337725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337698 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.337944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337915 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.338052 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.337975 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f602e80-772b-439c-a04c-526955a37d83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.338993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.338970 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f602e80-772b-439c-a04c-526955a37d83-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.339141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.339122 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.339245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.339226 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.339519 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.339503 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f602e80-772b-439c-a04c-526955a37d83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.390452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.390393 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f602e80-772b-439c-a04c-526955a37d83-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.390638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.390618 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwwj\" (UniqueName: \"kubernetes.io/projected/1f602e80-772b-439c-a04c-526955a37d83-kube-api-access-qmwwj\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7cxs\" (UID: \"1f602e80-772b-439c-a04c-526955a37d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.391028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.391013 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.391233 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.391211 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccchn\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn\") pod \"maas-default-gateway-openshift-default-7df854bf67-dkb6j\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.458693 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.458660 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:47.502962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.502747 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:47.697031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.696954 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:47.859170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.859133 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:47.926789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.926755 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j"] Nov 01 12:45:47.931510 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:47.931480 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106511c1_d902_40c3_a306_a5b8790400e4.slice/crio-4d99ad21b44ad895acfbcd9b6fb624cd45668053c716543b6a1f7530506ee45f WatchSource:0}: Error finding container 4d99ad21b44ad895acfbcd9b6fb624cd45668053c716543b6a1f7530506ee45f: Status 404 returned error can't find the container with id 4d99ad21b44ad895acfbcd9b6fb624cd45668053c716543b6a1f7530506ee45f Nov 01 12:45:47.934174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.934139 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:47.934262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.934206 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:47.934262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.934238 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:47.956013 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:45:47.955987 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f602e80_772b_439c_a04c_526955a37d83.slice/crio-231880c14192c493401b712f691ca11e77ee0951f4a9cb48038f6feb18d408c5 WatchSource:0}: Error finding container 231880c14192c493401b712f691ca11e77ee0951f4a9cb48038f6feb18d408c5: Status 404 returned error can't find the container with id 231880c14192c493401b712f691ca11e77ee0951f4a9cb48038f6feb18d408c5 Nov 01 12:45:47.958546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.958513 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:47.958613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.958581 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:47.958654 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.958611 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:45:47.973629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:47.973609 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs"] Nov 01 12:45:48.071780 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.071746 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" event={"ID":"1f602e80-772b-439c-a04c-526955a37d83","Type":"ContainerStarted","Data":"5adb8a97332543eb56c5a8cb3098b639a795d7dfc16b6c69975be5e858aa7aa5"} Nov 01 12:45:48.071947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.071802 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" event={"ID":"1f602e80-772b-439c-a04c-526955a37d83","Type":"ContainerStarted","Data":"231880c14192c493401b712f691ca11e77ee0951f4a9cb48038f6feb18d408c5"} Nov 01 12:45:48.074598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.074574 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" event={"ID":"106511c1-d902-40c3-a306-a5b8790400e4","Type":"ContainerStarted","Data":"030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d"} Nov 01 12:45:48.074748 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.074614 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" event={"ID":"106511c1-d902-40c3-a306-a5b8790400e4","Type":"ContainerStarted","Data":"4d99ad21b44ad895acfbcd9b6fb624cd45668053c716543b6a1f7530506ee45f"} Nov 01 12:45:48.115291 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.115256 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:48.190393 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.190341 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podStartSLOduration=1.1903257250000001 podStartE2EDuration="1.190325725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:48.187007806 +0000 UTC m=+1417.936950784" watchObservedRunningTime="2026-11-01 12:45:48.190325725 +0000 UTC m=+1417.940268764" Nov 01 12:45:48.309124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.309072 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podStartSLOduration=1.30905873 podStartE2EDuration="1.30905873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:48.306370968 +0000 UTC m=+1418.056313947" watchObservedRunningTime="2026-11-01 12:45:48.30905873 +0000 UTC m=+1418.059001708" Nov 01 12:45:48.459392 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.459360 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:48.502942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.502899 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:48.555257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.555160 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:48.981344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:48.981117 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:49.465705 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:49.464642 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:49.465705 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:49.464706 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:49.505022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:49.504991 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:49.505174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:49.505055 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:50.461324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:50.460699 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:50.461324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:50.460759 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:50.503222 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:50.503185 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:50.503644 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:50.503257 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:51.461325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:51.460694 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:51.461325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:51.460766 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:51.503795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:51.503760 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:51.504188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:51.503814 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:52.472325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:52.455800 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:52.472625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:52.462662 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:52.472791 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:52.472768 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:52.504085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:52.504053 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:52.504475 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:52.504104 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:53.465324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:53.463929 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:53.465324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:53.464012 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:53.503164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:53.503134 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:53.503372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:53.503181 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:54.460060 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:54.460020 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:54.460470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:54.460096 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:54.503749 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:54.503715 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:54.503894 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:54.503786 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:54.565284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:54.565251 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:55.066584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.066528 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:55.279823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.279776 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:55.460099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.460015 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 12:45:55.460099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.460078 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 12:45:55.503389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.503357 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 12:45:55.503517 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.503411 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 12:45:55.662461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:55.662427 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:45:56.678394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:56.678361 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:56.679922 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:56.679898 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:56.680175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:56.680157 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:45:56.750453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:56.750426 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:56.750958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:56.750929 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:56.751765 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:56.751749 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" Nov 01 12:45:59.945136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:59.945099 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h"] Nov 01 12:45:59.951372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:45:59.951349 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.028642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.028609 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.028838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.028661 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.028916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.028889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dxw\" (UniqueName: \"kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.028983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.028919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.048109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.047998 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-78de6246-serving-cert\"" Nov 01 12:46:00.056954 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.052355 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h"] Nov 01 12:46:00.130360 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.130317 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.130541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.130398 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.130676 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:46:00.130563 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-serving-cert: secret "maas-api-e2e-ait-78de6246-serving-cert" not found Nov 01 12:46:00.130676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.130639 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dxw\" (UniqueName: \"kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.130676 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:46:00.130671 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls podName:4b553037-edd3-430d-88b3-7f7d7a9e4395 nodeName:}" failed. No retries permitted until 2026-11-01 12:46:00.630649174 +0000 UTC m=+1430.380592136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls") pod "maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" (UID: "4b553037-edd3-430d-88b3-7f7d7a9e4395") : secret "maas-api-e2e-ait-78de6246-serving-cert" not found Nov 01 12:46:00.130899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.130730 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.131412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.131387 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.133930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.133899 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.186166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.186133 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dxw\" (UniqueName: \"kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.198931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.198866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:00.324867 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:46:00.324829 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:46:00.407411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.407378 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:00.578351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.576365 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:46:00.588348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.588036 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:00.665643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.665610 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.668445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.668417 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\") pod \"maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:00.865265 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:00.865176 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:01.337868 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:01.337832 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h"] Nov 01 12:46:02.223569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.223528 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" event={"ID":"4b553037-edd3-430d-88b3-7f7d7a9e4395","Type":"ContainerStarted","Data":"73e75ace706a5024c11686f0ab178a012c8f21f061dbbd9bab59c572705c5a65"} Nov 01 12:46:02.365291 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.356516 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7"] Nov 01 12:46:02.411467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.411435 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518349 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518458 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518525 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jl8\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518598 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518761 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518799 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518916 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.551358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.518958 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.632323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.607402 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7"] Nov 01 12:46:02.647505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.647470 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.649033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.649003 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.651704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.651684 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.652231 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.652209 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.653216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.649876 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.654115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.651629 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.654266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.652624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.654549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.653378 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.654761 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.654740 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.656272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.656251 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.657094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.657071 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.657642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.657621 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jl8\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.658142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.654057 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.660533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.660513 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.662215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.662105 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.923863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.923822 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.936678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.936604 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jl8\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:02.971320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:02.968550 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gdbv7\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:03.153802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.153587 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:03.341345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.341273 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:03.516400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.516337 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" podStartSLOduration=3.267577067 podStartE2EDuration="4.51631607s" totalImagesPullingTime="1.248739003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:46:03.514619499 +0000 UTC m=+1433.264562479" watchObservedRunningTime="2026-11-01 12:46:03.51631607 +0000 UTC m=+1433.266259047" Nov 01 12:46:03.793638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.779919 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:03.793638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.780101 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:03.793638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.780149 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:03.793638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:03.789444 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7"] Nov 01 12:46:04.116364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:04.115151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:04.300377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:04.299417 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:04.356366 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:04.355269 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" event={"ID":"1bea2e7e-a9c6-46b0-b930-8c769d293f66","Type":"ContainerStarted","Data":"a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1"} Nov 01 12:46:04.356366 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:04.355352 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" event={"ID":"1bea2e7e-a9c6-46b0-b930-8c769d293f66","Type":"ContainerStarted","Data":"450f25c3245e0e2188625a9b0123c49a9a8187153d1cd547b9d948daa813bc5f"} Nov 01 12:46:04.359069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:04.358261 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" event={"ID":"4b553037-edd3-430d-88b3-7f7d7a9e4395","Type":"ContainerStarted","Data":"72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9"} Nov 01 12:46:04.500462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:04.500402 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podStartSLOduration=2.500381762 podStartE2EDuration="2.500381762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:46:04.495039356 +0000 UTC m=+1434.244982335" watchObservedRunningTime="2026-11-01 12:46:04.500381762 +0000 UTC m=+1434.250324742" Nov 01 12:46:05.165360 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:05.161810 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:05.177434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:05.167635 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:05.177434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:05.167694 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:06.155127 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:06.154658 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:06.155127 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:06.154723 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:06.736462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:06.736429 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:06.959790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:06.959755 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:07.095705 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:07.095581 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:07.163363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:07.157727 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:07.163363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:07.157790 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:07.804560 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:07.804517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:08.161378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:08.159682 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:08.161378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:08.159745 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:09.155665 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:09.155625 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:09.156241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:09.156209 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:09.322077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:09.322040 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:10.162325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:10.160624 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:10.162325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:10.160686 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:10.480347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:10.480263 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:46:11.155067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:11.155031 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:11.155267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:11.155101 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:12.154559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:12.154519 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gdbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 12:46:12.154938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:12.154602 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 12:46:12.996545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:12.996514 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:13.236396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:13.236360 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:13.236824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:13.236804 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:13.237596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:13.237578 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:46:15.506118 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:15.506085 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:18.331166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:18.331101 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:18.508234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:18.508193 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:18.636526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:18.636422 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:23.883777 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:23.883741 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:29.223032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:29.222997 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:30.254293 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:46:30.254255 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:46:34.671157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:34.671123 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:40.126487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:40.126450 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:45.639485 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:45.639452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:50.911099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.911064 4484 generic.go:356] "Generic (PLEG): container finished" podID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerID="95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82" exitCode=0 Nov 01 12:46:50.911697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.911141 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" event={"ID":"59b768de-eab0-41ed-b2ee-4a7baf0122de","Type":"ContainerDied","Data":"95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82"} Nov 01 12:46:50.911773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.911743 4484 scope.go:122] "RemoveContainer" containerID="95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82" Nov 01 12:46:50.913057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.913034 4484 generic.go:356] "Generic (PLEG): container finished" podID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerID="61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d" exitCode=0 Nov 01 12:46:50.913173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.913068 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" event={"ID":"bc69f0c7-b732-4628-94ba-bf43ee87cee6","Type":"ContainerDied","Data":"61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d"} Nov 01 12:46:50.913563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.913542 4484 scope.go:122] "RemoveContainer" containerID="61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d" Nov 01 12:46:50.913662 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.913628 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:50.913721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.913699 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:50.913776 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.913739 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:50.915217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.915186 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:50.915331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.915259 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:50.915396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:50.915332 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:46:51.020690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:51.020661 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:51.861470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:51.861419 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:46:51.922076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:51.922042 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" event={"ID":"59b768de-eab0-41ed-b2ee-4a7baf0122de","Type":"ContainerStarted","Data":"27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79"} Nov 01 12:46:51.924098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:51.924070 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" event={"ID":"bc69f0c7-b732-4628-94ba-bf43ee87cee6","Type":"ContainerStarted","Data":"26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9"} Nov 01 12:46:52.419671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.419635 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:46:52.421061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.421037 4484 patch_prober.go:29] interesting pod/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 12:46:52.421177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.421087 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 12:46:52.702633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.702547 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:46:52.703615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.703574 4484 patch_prober.go:29] interesting pod/e2e-ait-78de6246-openshift-default-b77d885-9w225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 12:46:52.703708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.703640 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 12:46:52.862257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:52.862222 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:46:53.419579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:53.419543 4484 patch_prober.go:29] interesting pod/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 12:46:53.419958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:53.419613 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 12:46:53.702617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:53.702511 4484 patch_prober.go:29] interesting pod/e2e-ait-78de6246-openshift-default-b77d885-9w225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 12:46:53.702835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:53.702811 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 12:46:54.420187 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:54.420151 4484 patch_prober.go:29] interesting pod/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 12:46:54.420635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:54.420212 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 12:46:54.702483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:54.702393 4484 patch_prober.go:29] interesting pod/e2e-ait-78de6246-openshift-default-b77d885-9w225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 12:46:54.702483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:54.702458 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 12:46:55.420058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:55.420024 4484 patch_prober.go:29] interesting pod/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 12:46:55.420263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:55.420082 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 12:46:55.701903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:55.701811 4484 patch_prober.go:29] interesting pod/e2e-ait-78de6246-openshift-default-b77d885-9w225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 12:46:55.701903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:55.701876 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 12:46:56.420044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:56.420008 4484 patch_prober.go:29] interesting pod/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 12:46:56.420243 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:56.420073 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 12:46:56.494667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:56.494640 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:46:56.702377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:56.701950 4484 patch_prober.go:29] interesting pod/e2e-ait-78de6246-openshift-default-b77d885-9w225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 12:46:56.702377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:56.702020 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 12:46:57.452345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:57.452283 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:46:57.453568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:57.453547 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:46:57.707028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:57.706952 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:46:57.707424 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:57.707349 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:46:57.707962 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:46:57.707941 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:47:00.253457 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:00.253415 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:47:01.809912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:01.809875 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:07.208074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:07.208035 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:08.368907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:08.368874 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:10.154881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:10.154850 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:13.223696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:13.223663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:14.120105 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:14.120069 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:47:19.136957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:19.136922 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:23.060375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:23.060345 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l"] Nov 01 12:47:23.158795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:23.158759 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv"] Nov 01 12:47:23.402076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:23.401993 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:23.535226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:23.535185 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:25.102441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:25.102402 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:25.818738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:25.818705 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n"] Nov 01 12:47:25.925144 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:25.925094 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7"] Nov 01 12:47:26.170135 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:26.170042 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:26.306861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:26.306819 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:28.061363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.061281 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" podUID="90fda319-5fdc-404e-b79a-cafa8d1a0e65" containerName="payload-processing" containerID="cri-o://5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8" gracePeriod=25 Nov 01 12:47:28.110874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.110844 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:28.159786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.159731 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" podUID="68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" containerName="payload-pre-processing" containerID="cri-o://3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea" gracePeriod=25 Nov 01 12:47:28.359246 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.359184 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:47:28.444262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.444236 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n\" (UniqueName: \"kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n\") pod \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " Nov 01 12:47:28.444416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.444296 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume\") pod \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\" (UID: \"90fda319-5fdc-404e-b79a-cafa8d1a0e65\") " Nov 01 12:47:28.444416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.444365 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:47:28.444709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.444683 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume" pod "90fda319-5fdc-404e-b79a-cafa8d1a0e65" (UID: "90fda319-5fdc-404e-b79a-cafa8d1a0e65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:47:28.444822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.444808 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fda319-5fdc-404e-b79a-cafa8d1a0e65-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:28.446474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.446455 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n" pod "90fda319-5fdc-404e-b79a-cafa8d1a0e65" (UID: "90fda319-5fdc-404e-b79a-cafa8d1a0e65"). InnerVolumeSpecName "kube-api-access-75n4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:47:28.515375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.515344 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h"] Nov 01 12:47:28.526157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.526127 4484 generic.go:356] "Generic (PLEG): container finished" podID="68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" containerID="3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea" exitCode=0 Nov 01 12:47:28.526344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.526173 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" Nov 01 12:47:28.526344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.526208 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" event={"ID":"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf","Type":"ContainerDied","Data":"3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea"} Nov 01 12:47:28.526344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.526260 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv" event={"ID":"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf","Type":"ContainerDied","Data":"7f58c815bfec71a3369974c16790fa825c6e651a4e9d8ba58e4becbe33b1a589"} Nov 01 12:47:28.526344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.526295 4484 scope.go:122] "RemoveContainer" containerID="3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea" Nov 01 12:47:28.527755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.527734 4484 generic.go:356] "Generic (PLEG): container finished" podID="90fda319-5fdc-404e-b79a-cafa8d1a0e65" containerID="5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8" exitCode=0 Nov 01 12:47:28.527845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.527767 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" Nov 01 12:47:28.527845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.527823 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" event={"ID":"90fda319-5fdc-404e-b79a-cafa8d1a0e65","Type":"ContainerDied","Data":"5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8"} Nov 01 12:47:28.527942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.527877 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l" event={"ID":"90fda319-5fdc-404e-b79a-cafa8d1a0e65","Type":"ContainerDied","Data":"ff79431f9c9ac7f53c5fa134b8155c3a7055b9feb29c0389ed2ba9627043aa90"} Nov 01 12:47:28.537245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.537225 4484 scope.go:122] "RemoveContainer" containerID="3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea" Nov 01 12:47:28.537550 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:28.537530 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea\": container with ID starting with 3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea not found: ID does not exist" containerID="3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea" Nov 01 12:47:28.537618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.537559 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea"} err="failed to get container status \"3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea\": rpc error: code = NotFound desc = could not find container \"3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea\": container with ID starting with 3a5ce146e956599f2ed617bb9b56d5c705064f91e8d6e59e408568323a08a6ea not found: ID does not exist" Nov 01 12:47:28.537618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.537579 4484 scope.go:122] "RemoveContainer" containerID="5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8" Nov 01 12:47:28.546260 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.546236 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4\" (UniqueName: \"kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4\") pod \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " Nov 01 12:47:28.546395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.546378 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume\") pod \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\" (UID: \"68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf\") " Nov 01 12:47:28.546706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.546686 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume" pod "68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" (UID: "68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:47:28.547063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.547031 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75n4n\" (UniqueName: \"kubernetes.io/projected/90fda319-5fdc-404e-b79a-cafa8d1a0e65-kube-api-access-75n4n\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:28.547063 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.547054 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:28.547153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.547113 4484 scope.go:122] "RemoveContainer" containerID="5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8" Nov 01 12:47:28.547408 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:28.547389 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8\": container with ID starting with 5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8 not found: ID does not exist" containerID="5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8" Nov 01 12:47:28.547464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.547414 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8"} err="failed to get container status \"5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8\": rpc error: code = NotFound desc = could not find container \"5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8\": container with ID starting with 5b2d86f74e8e53c864d4bfdcd4136905106ab5b14078ee84d7b4ef60ce0deba8 not found: ID does not exist" Nov 01 12:47:28.548233 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.548214 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4" pod "68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" (UID: "68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf"). InnerVolumeSpecName "kube-api-access-k2wf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:47:28.590793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.589345 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr"] Nov 01 12:47:28.648043 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.648011 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wf4\" (UniqueName: \"kubernetes.io/projected/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf-kube-api-access-k2wf4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:28.676359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.676326 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l"] Nov 01 12:47:28.682078 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.682050 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78de6246-57f7974cc9-rsg7l"] Nov 01 12:47:28.834160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.834083 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:28.989470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:28.969203 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:29.113440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:29.113402 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv"] Nov 01 12:47:29.129218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:29.128247 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78de6246-97b77c54c-f6hcv"] Nov 01 12:47:30.325321 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:30.317249 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:47:30.457408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:30.447373 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" path="/var/lib/kubelet/pods/68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf/volumes" Nov 01 12:47:30.457408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:30.448211 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fda319-5fdc-404e-b79a-cafa8d1a0e65" path="/var/lib/kubelet/pods/90fda319-5fdc-404e-b79a-cafa8d1a0e65/volumes" Nov 01 12:47:30.702095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:30.701997 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:30.830324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:30.820691 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" podUID="1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" containerName="payload-processing" containerID="cri-o://a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90" gracePeriod=25 Nov 01 12:47:30.925493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:30.925432 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" podUID="9e1fcba7-129d-4fdb-80f0-d152e7c72c01" containerName="payload-pre-processing" containerID="cri-o://5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50" gracePeriod=25 Nov 01 12:47:31.224941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.224915 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:47:31.253775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.253745 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26\" (UniqueName: \"kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26\") pod \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " Nov 01 12:47:31.253906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.253889 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume\") pod \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\" (UID: \"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0\") " Nov 01 12:47:31.254275 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.254251 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume" pod "1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" (UID: "1b1b75ef-a924-4f95-aaa9-355bc8a40cc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:47:31.254632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.254614 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:31.255987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.255964 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26" pod "1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" (UID: "1b1b75ef-a924-4f95-aaa9-355bc8a40cc0"). InnerVolumeSpecName "kube-api-access-dpr26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:47:31.256790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.256774 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:47:31.355657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.355553 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume\") pod \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " Nov 01 12:47:31.355657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.355633 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc\" (UniqueName: \"kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc\") pod \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\" (UID: \"9e1fcba7-129d-4fdb-80f0-d152e7c72c01\") " Nov 01 12:47:31.356196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.355981 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume" pod "9e1fcba7-129d-4fdb-80f0-d152e7c72c01" (UID: "9e1fcba7-129d-4fdb-80f0-d152e7c72c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:47:31.356621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.356594 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpr26\" (UniqueName: \"kubernetes.io/projected/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0-kube-api-access-dpr26\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:31.356733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.356624 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:31.357831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.357807 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc" pod "9e1fcba7-129d-4fdb-80f0-d152e7c72c01" (UID: "9e1fcba7-129d-4fdb-80f0-d152e7c72c01"). InnerVolumeSpecName "kube-api-access-r98sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:47:31.458747 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.458707 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r98sc\" (UniqueName: \"kubernetes.io/projected/9e1fcba7-129d-4fdb-80f0-d152e7c72c01-kube-api-access-r98sc\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:31.595016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.594984 4484 generic.go:356] "Generic (PLEG): container finished" podID="1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" containerID="a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90" exitCode=0 Nov 01 12:47:31.595209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.595039 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" Nov 01 12:47:31.595209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.595046 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" event={"ID":"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0","Type":"ContainerDied","Data":"a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90"} Nov 01 12:47:31.595209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.595099 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n" event={"ID":"1b1b75ef-a924-4f95-aaa9-355bc8a40cc0","Type":"ContainerDied","Data":"d86672c3799c81d4f93d3b1483122bc64c3a2fc631d1eed945fbb4d15b3baef2"} Nov 01 12:47:31.595209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.595118 4484 scope.go:122] "RemoveContainer" containerID="a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90" Nov 01 12:47:31.596568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.596548 4484 generic.go:356] "Generic (PLEG): container finished" podID="9e1fcba7-129d-4fdb-80f0-d152e7c72c01" containerID="5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50" exitCode=0 Nov 01 12:47:31.596655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.596587 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" Nov 01 12:47:31.596714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.596678 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" event={"ID":"9e1fcba7-129d-4fdb-80f0-d152e7c72c01","Type":"ContainerDied","Data":"5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50"} Nov 01 12:47:31.596758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.596728 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7" event={"ID":"9e1fcba7-129d-4fdb-80f0-d152e7c72c01","Type":"ContainerDied","Data":"351b9316134b7b94ed4d628396aeeccb25de4e46209460558ba48442406963e2"} Nov 01 12:47:31.608160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.608139 4484 scope.go:122] "RemoveContainer" containerID="a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90" Nov 01 12:47:31.608482 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:31.608455 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90\": container with ID starting with a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90 not found: ID does not exist" containerID="a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90" Nov 01 12:47:31.608574 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.608489 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90"} err="failed to get container status \"a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90\": rpc error: code = NotFound desc = could not find container \"a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90\": container with ID starting with a6de1ef55175e0d7455dc57d1c090da1a3e7355f08b7c9ed18916bcbe1005f90 not found: ID does not exist" Nov 01 12:47:31.608574 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.608506 4484 scope.go:122] "RemoveContainer" containerID="5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50" Nov 01 12:47:31.618126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.618107 4484 scope.go:122] "RemoveContainer" containerID="5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50" Nov 01 12:47:31.618412 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:31.618395 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50\": container with ID starting with 5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50 not found: ID does not exist" containerID="5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50" Nov 01 12:47:31.618487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.618419 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50"} err="failed to get container status \"5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50\": rpc error: code = NotFound desc = could not find container \"5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50\": container with ID starting with 5fdc69ac426fdf2450c80acaa44024af3538adf13848fb6cba6478cebd796d50 not found: ID does not exist" Nov 01 12:47:31.749579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.749538 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7"] Nov 01 12:47:31.753469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.753445 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b2ccbb55-cc657cfcf-gdqg7"] Nov 01 12:47:31.853261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.853197 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n"] Nov 01 12:47:31.857669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:31.857609 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b2ccbb55-6ddc899b47-z4f6n"] Nov 01 12:47:32.371388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:32.371357 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" path="/var/lib/kubelet/pods/1b1b75ef-a924-4f95-aaa9-355bc8a40cc0/volumes" Nov 01 12:47:32.371851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:32.371836 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1fcba7-129d-4fdb-80f0-d152e7c72c01" path="/var/lib/kubelet/pods/9e1fcba7-129d-4fdb-80f0-d152e7c72c01/volumes" Nov 01 12:47:33.516105 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.516066 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" podUID="49178b94-1035-4efd-a9c5-fc616bd00e77" containerName="payload-processing" containerID="cri-o://2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee" gracePeriod=25 Nov 01 12:47:33.590477 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.590427 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" podUID="6623ea9d-984a-4597-8fba-d27479f83fc7" containerName="payload-pre-processing" containerID="cri-o://825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887" gracePeriod=25 Nov 01 12:47:33.778387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.777822 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:33.877627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.877600 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:47:33.914153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.914133 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:47:33.995548 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.995494 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume\") pod \"49178b94-1035-4efd-a9c5-fc616bd00e77\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " Nov 01 12:47:33.995702 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.995660 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume\") pod \"6623ea9d-984a-4597-8fba-d27479f83fc7\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " Nov 01 12:47:33.995769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.995706 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg\" (UniqueName: \"kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg\") pod \"6623ea9d-984a-4597-8fba-d27479f83fc7\" (UID: \"6623ea9d-984a-4597-8fba-d27479f83fc7\") " Nov 01 12:47:33.995769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.995749 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2\" (UniqueName: \"kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2\") pod \"49178b94-1035-4efd-a9c5-fc616bd00e77\" (UID: \"49178b94-1035-4efd-a9c5-fc616bd00e77\") " Nov 01 12:47:33.996365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.996336 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume" pod "49178b94-1035-4efd-a9c5-fc616bd00e77" (UID: "49178b94-1035-4efd-a9c5-fc616bd00e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:47:33.996687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.996498 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49178b94-1035-4efd-a9c5-fc616bd00e77-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:33.996965 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.996943 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume" pod "6623ea9d-984a-4597-8fba-d27479f83fc7" (UID: "6623ea9d-984a-4597-8fba-d27479f83fc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:47:33.998192 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.998169 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg" pod "6623ea9d-984a-4597-8fba-d27479f83fc7" (UID: "6623ea9d-984a-4597-8fba-d27479f83fc7"). InnerVolumeSpecName "kube-api-access-mhhhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:47:33.998817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:33.998795 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2" pod "49178b94-1035-4efd-a9c5-fc616bd00e77" (UID: "49178b94-1035-4efd-a9c5-fc616bd00e77"). InnerVolumeSpecName "kube-api-access-rk6t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:47:34.097911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.097830 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6623ea9d-984a-4597-8fba-d27479f83fc7-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:34.097911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.097858 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhhg\" (UniqueName: \"kubernetes.io/projected/6623ea9d-984a-4597-8fba-d27479f83fc7-kube-api-access-mhhhg\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:34.097911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.097868 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6t2\" (UniqueName: \"kubernetes.io/projected/49178b94-1035-4efd-a9c5-fc616bd00e77-kube-api-access-rk6t2\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:47:34.620712 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.620675 4484 generic.go:356] "Generic (PLEG): container finished" podID="6623ea9d-984a-4597-8fba-d27479f83fc7" containerID="825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887" exitCode=0 Nov 01 12:47:34.621150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.620723 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" Nov 01 12:47:34.621150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.620761 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" event={"ID":"6623ea9d-984a-4597-8fba-d27479f83fc7","Type":"ContainerDied","Data":"825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887"} Nov 01 12:47:34.621150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.620832 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr" event={"ID":"6623ea9d-984a-4597-8fba-d27479f83fc7","Type":"ContainerDied","Data":"781a8e8fd710c95d4b2e926244f221f27c5fdae64c44a62c32d4ae05a871f03e"} Nov 01 12:47:34.621150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.620866 4484 scope.go:122] "RemoveContainer" containerID="825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887" Nov 01 12:47:34.622546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.622521 4484 generic.go:356] "Generic (PLEG): container finished" podID="49178b94-1035-4efd-a9c5-fc616bd00e77" containerID="2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee" exitCode=0 Nov 01 12:47:34.622688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.622563 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" Nov 01 12:47:34.622688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.622575 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" event={"ID":"49178b94-1035-4efd-a9c5-fc616bd00e77","Type":"ContainerDied","Data":"2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee"} Nov 01 12:47:34.622688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.622614 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h" event={"ID":"49178b94-1035-4efd-a9c5-fc616bd00e77","Type":"ContainerDied","Data":"f42a858f8349e578d02890a4a56caa46779f94bd003f97b29f060705c1161e21"} Nov 01 12:47:34.631628 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.631592 4484 scope.go:122] "RemoveContainer" containerID="825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887" Nov 01 12:47:34.631874 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:34.631855 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887\": container with ID starting with 825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887 not found: ID does not exist" containerID="825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887" Nov 01 12:47:34.631944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.631886 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887"} err="failed to get container status \"825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887\": rpc error: code = NotFound desc = could not find container \"825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887\": container with ID starting with 825216ec797a069ee2a9a714177f85a3a072d6f8f857dff711b53e22527b4887 not found: ID does not exist" Nov 01 12:47:34.631944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.631909 4484 scope.go:122] "RemoveContainer" containerID="2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee" Nov 01 12:47:34.640789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.640770 4484 scope.go:122] "RemoveContainer" containerID="2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee" Nov 01 12:47:34.641066 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:47:34.641047 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee\": container with ID starting with 2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee not found: ID does not exist" containerID="2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee" Nov 01 12:47:34.641119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.641071 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee"} err="failed to get container status \"2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee\": rpc error: code = NotFound desc = could not find container \"2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee\": container with ID starting with 2b755bff48720c8161b049bf54ab900916bf92891cb5f74ab7a6076cdb57a8ee not found: ID does not exist" Nov 01 12:47:34.773802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.773773 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr"] Nov 01 12:47:34.819930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.819903 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e4e-7c844796f9-s9dvr"] Nov 01 12:47:34.879342 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.879295 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h"] Nov 01 12:47:34.886396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:34.886328 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e4e-84b88b45c7-md88h"] Nov 01 12:47:35.981938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:35.981867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:36.371327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:36.371233 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49178b94-1035-4efd-a9c5-fc616bd00e77" path="/var/lib/kubelet/pods/49178b94-1035-4efd-a9c5-fc616bd00e77/volumes" Nov 01 12:47:36.371726 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:36.371711 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6623ea9d-984a-4597-8fba-d27479f83fc7" path="/var/lib/kubelet/pods/6623ea9d-984a-4597-8fba-d27479f83fc7/volumes" Nov 01 12:47:39.616567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:39.616533 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:40.220355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:40.219056 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:40.700270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:40.700156 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:41.486992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:41.486959 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:43.079276 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:43.079247 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:45.065644 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:45.065606 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:45.276849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:45.276810 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:45.404470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:45.404388 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:47.186701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:47.186672 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:48.401451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:48.401413 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:48.591447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:48.591411 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.012760 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85"] Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013875 4484 cpu_manager.go:414] "containerMap: removing container" podUID="90fda319-5fdc-404e-b79a-cafa8d1a0e65" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013897 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fda319-5fdc-404e-b79a-cafa8d1a0e65" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013922 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6623ea9d-984a-4597-8fba-d27479f83fc7" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013930 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6623ea9d-984a-4597-8fba-d27479f83fc7" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013956 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9e1fcba7-129d-4fdb-80f0-d152e7c72c01" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013964 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1fcba7-129d-4fdb-80f0-d152e7c72c01" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013980 4484 cpu_manager.go:414] "containerMap: removing container" podUID="49178b94-1035-4efd-a9c5-fc616bd00e77" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.013987 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49178b94-1035-4efd-a9c5-fc616bd00e77" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014009 4484 cpu_manager.go:414] "containerMap: removing container" podUID="68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014016 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014050 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014056 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014204 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6623ea9d-984a-4597-8fba-d27479f83fc7" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014219 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1fcba7-129d-4fdb-80f0-d152e7c72c01" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014230 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="49178b94-1035-4efd-a9c5-fc616bd00e77" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014240 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ee3cfd-3cf7-4ad2-b766-0b9e2bdae1bf" containerName="payload-pre-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014253 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1b75ef-a924-4f95-aaa9-355bc8a40cc0" containerName="payload-processing" Nov 01 12:47:49.015329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.014265 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fda319-5fdc-404e-b79a-cafa8d1a0e65" containerName="payload-processing" Nov 01 12:47:49.023976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.023837 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.054616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.054576 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85"] Nov 01 12:47:49.085202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.085174 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-6c650b-dockercfg-5x4nr\"" Nov 01 12:47:49.085558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.085539 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-6c650b\"" Nov 01 12:47:49.119857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.119398 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.119857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.119560 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnn7\" (UniqueName: \"kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7\") pod \"payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.219443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.219401 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc"] Nov 01 12:47:49.285178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.285098 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.338344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.338269 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.339167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.339139 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnn7\" (UniqueName: \"kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7\") pod \"payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.344483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.344462 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc"] Nov 01 12:47:49.344783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.344727 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.407527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.407490 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnn7\" (UniqueName: \"kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7\") pod \"payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.440402 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.440369 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.440639 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.440615 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvtst\" (UniqueName: \"kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst\") pod \"payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.542371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.542270 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvtst\" (UniqueName: \"kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst\") pod \"payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.542533 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.542494 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.543257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.543229 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.595499 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.595475 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvtst\" (UniqueName: \"kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst\") pod \"payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:49.672633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.672597 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:49.695528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:49.695503 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:50.059280 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.059208 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85"] Nov 01 12:47:50.065590 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:47:50.065564 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7140dee5_84f0_4805_962f_f22bffa8f545.slice/crio-fba5ccea41fd8b3cc94c14b89e8e2cc3650372c181d9f572f0b20ddf2fc05e7c WatchSource:0}: Error finding container fba5ccea41fd8b3cc94c14b89e8e2cc3650372c181d9f572f0b20ddf2fc05e7c: Status 404 returned error can't find the container with id fba5ccea41fd8b3cc94c14b89e8e2cc3650372c181d9f572f0b20ddf2fc05e7c Nov 01 12:47:50.110408 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.110381 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc"] Nov 01 12:47:50.115522 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:47:50.115489 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80896d05_d706_4abb_a93a_7cd061b4e36e.slice/crio-0eacba522b2a764f22b87f384a65e875a894216405f0571cc5d96d4d24dbc6bf WatchSource:0}: Error finding container 0eacba522b2a764f22b87f384a65e875a894216405f0571cc5d96d4d24dbc6bf: Status 404 returned error can't find the container with id 0eacba522b2a764f22b87f384a65e875a894216405f0571cc5d96d4d24dbc6bf Nov 01 12:47:50.801500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.801457 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" event={"ID":"7140dee5-84f0-4805-962f-f22bffa8f545","Type":"ContainerStarted","Data":"1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b"} Nov 01 12:47:50.802007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.801525 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" event={"ID":"7140dee5-84f0-4805-962f-f22bffa8f545","Type":"ContainerStarted","Data":"fba5ccea41fd8b3cc94c14b89e8e2cc3650372c181d9f572f0b20ddf2fc05e7c"} Nov 01 12:47:50.802507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.802486 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:47:50.804367 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.804345 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" event={"ID":"80896d05-d706-4abb-a93a-7cd061b4e36e","Type":"ContainerStarted","Data":"4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9"} Nov 01 12:47:50.804453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.804391 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" event={"ID":"80896d05-d706-4abb-a93a-7cd061b4e36e","Type":"ContainerStarted","Data":"0eacba522b2a764f22b87f384a65e875a894216405f0571cc5d96d4d24dbc6bf"} Nov 01 12:47:50.804453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.804434 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:47:50.910468 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:50.910424 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" podStartSLOduration=2.910410979 podStartE2EDuration="2.910410979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:47:50.907468821 +0000 UTC m=+1540.657411811" watchObservedRunningTime="2026-11-01 12:47:50.910410979 +0000 UTC m=+1540.660353985" Nov 01 12:47:51.014651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:51.014602 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" podStartSLOduration=2.01458855 podStartE2EDuration="2.01458855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:47:51.011278542 +0000 UTC m=+1540.761221521" watchObservedRunningTime="2026-11-01 12:47:51.01458855 +0000 UTC m=+1540.764531527" Nov 01 12:47:52.989530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:52.989497 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:47:58.316030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:47:58.315988 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:00.321682 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:00.321631 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:48:01.816051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:01.816011 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:48:02.821438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:02.821412 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:48:03.146886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.146809 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h"] Nov 01 12:48:03.147056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.147036 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" podUID="4b553037-edd3-430d-88b3-7f7d7a9e4395" containerName="maas-api" containerID="cri-o://72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9" gracePeriod=30 Nov 01 12:48:03.242065 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.242034 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:03.509507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.501505 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:48:03.562401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.562369 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw\" (UniqueName: \"kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw\") pod \"4b553037-edd3-430d-88b3-7f7d7a9e4395\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " Nov 01 12:48:03.562579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.562424 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs\") pod \"4b553037-edd3-430d-88b3-7f7d7a9e4395\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " Nov 01 12:48:03.562633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.562602 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\") pod \"4b553037-edd3-430d-88b3-7f7d7a9e4395\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " Nov 01 12:48:03.562676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.562642 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca\") pod \"4b553037-edd3-430d-88b3-7f7d7a9e4395\" (UID: \"4b553037-edd3-430d-88b3-7f7d7a9e4395\") " Nov 01 12:48:03.563628 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.563595 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca" pod "4b553037-edd3-430d-88b3-7f7d7a9e4395" (UID: "4b553037-edd3-430d-88b3-7f7d7a9e4395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:48:03.565629 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.565592 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw" pod "4b553037-edd3-430d-88b3-7f7d7a9e4395" (UID: "4b553037-edd3-430d-88b3-7f7d7a9e4395"). InnerVolumeSpecName "kube-api-access-24dxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:48:03.565981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.565959 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs" pod "4b553037-edd3-430d-88b3-7f7d7a9e4395" (UID: "4b553037-edd3-430d-88b3-7f7d7a9e4395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:48:03.566834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.566809 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls" pod "4b553037-edd3-430d-88b3-7f7d7a9e4395" (UID: "4b553037-edd3-430d-88b3-7f7d7a9e4395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:48:03.665079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.664998 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dxw\" (UniqueName: \"kubernetes.io/projected/4b553037-edd3-430d-88b3-7f7d7a9e4395-kube-api-access-24dxw\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:03.665079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.665040 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:03.665079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.665058 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b553037-edd3-430d-88b3-7f7d7a9e4395-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:03.665079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.665073 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b553037-edd3-430d-88b3-7f7d7a9e4395-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:03.959099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.959015 4484 generic.go:356] "Generic (PLEG): container finished" podID="4b553037-edd3-430d-88b3-7f7d7a9e4395" containerID="72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9" exitCode=0 Nov 01 12:48:03.959099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.959079 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" Nov 01 12:48:03.959536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.959102 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" event={"ID":"4b553037-edd3-430d-88b3-7f7d7a9e4395","Type":"ContainerDied","Data":"72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9"} Nov 01 12:48:03.959536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.959161 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h" event={"ID":"4b553037-edd3-430d-88b3-7f7d7a9e4395","Type":"ContainerDied","Data":"73e75ace706a5024c11686f0ab178a012c8f21f061dbbd9bab59c572705c5a65"} Nov 01 12:48:03.959536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.959186 4484 scope.go:122] "RemoveContainer" containerID="72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9" Nov 01 12:48:03.970197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.970166 4484 scope.go:122] "RemoveContainer" containerID="72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9" Nov 01 12:48:03.970492 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:03.970472 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9\": container with ID starting with 72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9 not found: ID does not exist" containerID="72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9" Nov 01 12:48:03.970571 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:03.970502 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9"} err="failed to get container status \"72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9\": rpc error: code = NotFound desc = could not find container \"72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9\": container with ID starting with 72ec1c06689a6aeffdc77932f22c1991077de6f23a99dba3737b751f8ca804b9 not found: ID does not exist" Nov 01 12:48:04.111800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:04.111767 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h"] Nov 01 12:48:04.157937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:04.157903 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78de6246-5d48ccbc99-w2j2h"] Nov 01 12:48:04.408553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:04.408175 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b553037-edd3-430d-88b3-7f7d7a9e4395" path="/var/lib/kubelet/pods/4b553037-edd3-430d-88b3-7f7d7a9e4395/volumes" Nov 01 12:48:04.571023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:04.570993 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:05.010480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:05.010452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:05.340228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:05.340142 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:05.681374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:05.677699 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:05.800326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:05.797832 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225"] Nov 01 12:48:05.800326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:05.798162 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" containerID="cri-o://26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9" gracePeriod=30 Nov 01 12:48:06.332339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:06.325871 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:06.460798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:06.460758 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:06.589077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:06.589000 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:06.710337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:06.710288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:07.023021 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.022983 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:07.101489 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.101457 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh"] Nov 01 12:48:07.102245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.102224 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4b553037-edd3-430d-88b3-7f7d7a9e4395" containerName="maas-api" Nov 01 12:48:07.102245 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.102247 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b553037-edd3-430d-88b3-7f7d7a9e4395" containerName="maas-api" Nov 01 12:48:07.102488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.102471 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b553037-edd3-430d-88b3-7f7d7a9e4395" containerName="maas-api" Nov 01 12:48:07.107418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.107400 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.150445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.150409 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:07.155758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.155738 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-61ffb38d-gw-openshift-default-dockercfg-5bkgt\"" Nov 01 12:48:07.200921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.200880 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.200981 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201069 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201224 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201114 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k875d\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201313 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201398 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201496 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201637 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.201698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.201681 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.206261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.206236 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh"] Nov 01 12:48:07.323409 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.309860 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.332473 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.323864 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.333234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.333203 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.338719 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.338687 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.339003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.338984 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.345626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.345594 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.345941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.345910 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k875d\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.346242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.346222 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.352627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.352599 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.352963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.319817 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:07.353181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.312516 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.353414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.342707 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.366334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.364759 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.366830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.366802 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.367222 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.367202 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.402375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.402340 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.405199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.405157 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.419195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.419042 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.429669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.429597 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k875d\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d\") pod \"e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:07.467983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.467862 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:07.721150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:07.721066 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:08.154986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.154946 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh"] Nov 01 12:48:08.160073 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:48:08.160037 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a37a6db_f4b0_4ffe_b0be_1746671f12fa.slice/crio-03ba53250f395e77363bed042d18fa97c2150fee89f01bef2290562af8464444 WatchSource:0}: Error finding container 03ba53250f395e77363bed042d18fa97c2150fee89f01bef2290562af8464444: Status 404 returned error can't find the container with id 03ba53250f395e77363bed042d18fa97c2150fee89f01bef2290562af8464444 Nov 01 12:48:08.162718 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.162682 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:48:08.162794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.162753 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:48:08.162794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.162790 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:48:08.480846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.480817 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:08.740637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.740539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:08.759930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.759903 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm"] Nov 01 12:48:08.760109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:08.760086 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" containerID="cri-o://27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79" gracePeriod=30 Nov 01 12:48:09.125811 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.124117 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" event={"ID":"8a37a6db-f4b0-4ffe-b0be-1746671f12fa","Type":"ContainerStarted","Data":"b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4"} Nov 01 12:48:09.125811 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.124191 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" event={"ID":"8a37a6db-f4b0-4ffe-b0be-1746671f12fa","Type":"ContainerStarted","Data":"03ba53250f395e77363bed042d18fa97c2150fee89f01bef2290562af8464444"} Nov 01 12:48:09.241584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.241532 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podStartSLOduration=2.241519126 podStartE2EDuration="2.241519126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:09.237994442 +0000 UTC m=+1558.987937420" watchObservedRunningTime="2026-11-01 12:48:09.241519126 +0000 UTC m=+1558.991462104" Nov 01 12:48:09.442381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.431427 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:09.721627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.721541 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:09.727615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.727578 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:09.940286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.940252 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd"] Nov 01 12:48:09.946318 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:09.946273 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.038612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.038581 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e58e0934-openshift-default-dockercfg-xjgld\"" Nov 01 12:48:10.056337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.043561 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.092534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.062868 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:10.092973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.092491 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.093168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.093143 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.131323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.130628 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.131323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.130777 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.131579 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.131330 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.134878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.131732 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.137454 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.137427 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.143724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.143680 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrzt\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.267446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.267296 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:10.269186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.269159 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:48:10.286208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.286176 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.283011 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.379491 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.379736 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.379780 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrzt\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.379909 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.379948 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.380026 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.380085 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.380141 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.383335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.380627 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.390627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.390581 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.391133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.391113 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.442532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.396782 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.442532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.416003 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.461652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.461617 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.463425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.462710 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd"] Nov 01 12:48:10.469013 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.464276 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.472073 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.472045 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrzt\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt\") pod \"e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:10.582345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.582243 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:10.659338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.658491 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e58e0934-openshift-default-dockercfg-xjgld\"" Nov 01 12:48:10.661869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:10.661838 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:11.093215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.093184 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:48:11.181822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.181755 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd"] Nov 01 12:48:11.187598 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:48:11.187563 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51727984_f64c_422b_ba4c_c9c1d3f6d414.slice/crio-ba6036eb2f22c8ef005135a38701186ab6e31444e5d91770370b8b58abad3a7e WatchSource:0}: Error finding container ba6036eb2f22c8ef005135a38701186ab6e31444e5d91770370b8b58abad3a7e: Status 404 returned error can't find the container with id ba6036eb2f22c8ef005135a38701186ab6e31444e5d91770370b8b58abad3a7e Nov 01 12:48:11.193507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193439 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.193507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193501 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.193668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193603 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.193737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193719 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.193794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193734 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:48:11.193843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193794 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.193843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193825 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:48:11.193932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193861 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.193932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193877 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:48:11.194029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.193974 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.194029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194002 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:11.194124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194010 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.194124 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194074 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data\") pod \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\" (UID: \"bc69f0c7-b732-4628-94ba-bf43ee87cee6\") " Nov 01 12:48:11.194238 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194217 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:11.194583 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194562 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:48:11.194824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194803 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:11.194921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194901 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.194985 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194929 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.194985 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.194947 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.195131 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.195113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:11.195944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.195924 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:48:11.196057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.196041 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:48:11.196177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.196158 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:11.196828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.196808 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l" pod "bc69f0c7-b732-4628-94ba-bf43ee87cee6" (UID: "bc69f0c7-b732-4628-94ba-bf43ee87cee6"). InnerVolumeSpecName "kube-api-access-2ss4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:48:11.280374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.280004 4484 generic.go:356] "Generic (PLEG): container finished" podID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerID="26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9" exitCode=0 Nov 01 12:48:11.280374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.280086 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" Nov 01 12:48:11.280374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.280292 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" event={"ID":"bc69f0c7-b732-4628-94ba-bf43ee87cee6","Type":"ContainerDied","Data":"26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9"} Nov 01 12:48:11.280374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.280371 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225" event={"ID":"bc69f0c7-b732-4628-94ba-bf43ee87cee6","Type":"ContainerDied","Data":"74ef6c9871d5c2ee279522b3efac7251f2695cd2d3da2662080a8ef823911d66"} Nov 01 12:48:11.280670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.280410 4484 scope.go:122] "RemoveContainer" containerID="26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9" Nov 01 12:48:11.293081 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.293036 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" event={"ID":"51727984-f64c-422b-ba4c-c9c1d3f6d414","Type":"ContainerStarted","Data":"ba6036eb2f22c8ef005135a38701186ab6e31444e5d91770370b8b58abad3a7e"} Nov 01 12:48:11.296196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.296175 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.296278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.296197 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ss4l\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-kube-api-access-2ss4l\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.296278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.296207 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.296278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.296217 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.296278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.296229 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.296278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.296237 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc69f0c7-b732-4628-94ba-bf43ee87cee6-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:11.300405 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.300382 4484 scope.go:122] "RemoveContainer" containerID="61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d" Nov 01 12:48:11.310646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.310622 4484 scope.go:122] "RemoveContainer" containerID="26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9" Nov 01 12:48:11.310968 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:11.310941 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9\": container with ID starting with 26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9 not found: ID does not exist" containerID="26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9" Nov 01 12:48:11.311068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.310976 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9"} err="failed to get container status \"26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9\": rpc error: code = NotFound desc = could not find container \"26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9\": container with ID starting with 26a5c309ce80f468154d1a77a645077f2eeaae33f9d14d3e72961a9b9f9ff6c9 not found: ID does not exist" Nov 01 12:48:11.311068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.310994 4484 scope.go:122] "RemoveContainer" containerID="61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d" Nov 01 12:48:11.311289 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:11.311266 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d\": container with ID starting with 61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d not found: ID does not exist" containerID="61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d" Nov 01 12:48:11.311508 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.311293 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d"} err="failed to get container status \"61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d\": rpc error: code = NotFound desc = could not find container \"61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d\": container with ID starting with 61954436c7250e94e9cc1e12158f9f2af68782e3849642c053c27b5d2006945d not found: ID does not exist" Nov 01 12:48:11.437437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.437394 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225"] Nov 01 12:48:11.439758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.439728 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-78de6246-openshift-default-b77d885-9w225"] Nov 01 12:48:11.954137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:11.954103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:12.144195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.144157 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:12.301701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.301665 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" event={"ID":"51727984-f64c-422b-ba4c-c9c1d3f6d414","Type":"ContainerStarted","Data":"2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c"} Nov 01 12:48:12.370464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.370430 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" path="/var/lib/kubelet/pods/bc69f0c7-b732-4628-94ba-bf43ee87cee6/volumes" Nov 01 12:48:12.455627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.455591 4484 patch_prober.go:29] interesting pod/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:48:12.455792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.455668 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:48:12.503556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.503502 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podStartSLOduration=3.503488211 podStartE2EDuration="3.503488211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:12.454927443 +0000 UTC m=+1562.204870422" watchObservedRunningTime="2026-11-01 12:48:12.503488211 +0000 UTC m=+1562.253431243" Nov 01 12:48:12.623200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.623082 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:12.662135 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.662107 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:12.667435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:12.667411 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:13.391647 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:13.391614 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:13.392484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:13.392466 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:48:14.221112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.221080 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:48:14.290198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290168 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290247 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290338 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290390 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290372 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290482 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290489 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:14.290572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290529 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290573 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290617 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290772 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290710 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert\") pod \"59b768de-eab0-41ed-b2ee-4a7baf0122de\" (UID: \"59b768de-eab0-41ed-b2ee-4a7baf0122de\") " Nov 01 12:48:14.290835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290818 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:14.291156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.290936 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:14.291156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:14.291324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291289 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:48:14.291480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291461 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b768de-eab0-41ed-b2ee-4a7baf0122de-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.291549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291484 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.291549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291499 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.291549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291513 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.291549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.291526 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.292661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.292639 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:48:14.292740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.292687 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "kube-api-access-zqhwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:48:14.293016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.293001 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:48:14.293053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.293016 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token" pod "59b768de-eab0-41ed-b2ee-4a7baf0122de" (UID: "59b768de-eab0-41ed-b2ee-4a7baf0122de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:48:14.393592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.393559 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.393592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.393593 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.393994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.393609 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhwv\" (UniqueName: \"kubernetes.io/projected/59b768de-eab0-41ed-b2ee-4a7baf0122de-kube-api-access-zqhwv\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.393994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.393624 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b768de-eab0-41ed-b2ee-4a7baf0122de-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:48:14.404114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.404079 4484 generic.go:356] "Generic (PLEG): container finished" podID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerID="27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79" exitCode=0 Nov 01 12:48:14.404256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.404151 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" Nov 01 12:48:14.404256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.404158 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" event={"ID":"59b768de-eab0-41ed-b2ee-4a7baf0122de","Type":"ContainerDied","Data":"27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79"} Nov 01 12:48:14.404256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.404223 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm" event={"ID":"59b768de-eab0-41ed-b2ee-4a7baf0122de","Type":"ContainerDied","Data":"3d2b9c9fc736f3078a866fa745a1b36167cd6cc5579dc1dc427fd74d9b49358d"} Nov 01 12:48:14.404256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.404252 4484 scope.go:122] "RemoveContainer" containerID="27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79" Nov 01 12:48:14.416036 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.416016 4484 scope.go:122] "RemoveContainer" containerID="95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82" Nov 01 12:48:14.425601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.425566 4484 scope.go:122] "RemoveContainer" containerID="27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79" Nov 01 12:48:14.425917 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:14.425894 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79\": container with ID starting with 27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79 not found: ID does not exist" containerID="27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79" Nov 01 12:48:14.425991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.425926 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79"} err="failed to get container status \"27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79\": rpc error: code = NotFound desc = could not find container \"27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79\": container with ID starting with 27c180c23a26c563b3d811a104f1a393a53fd4fc9516be710e009a671b5d0d79 not found: ID does not exist" Nov 01 12:48:14.425991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.425959 4484 scope.go:122] "RemoveContainer" containerID="95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82" Nov 01 12:48:14.426273 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:14.426245 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82\": container with ID starting with 95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82 not found: ID does not exist" containerID="95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82" Nov 01 12:48:14.426371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.426283 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82"} err="failed to get container status \"95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82\": rpc error: code = NotFound desc = could not find container \"95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82\": container with ID starting with 95967dbea2e931524d496ea0eae68abde6b2618519e283480a1fbb3c6a3a4f82 not found: ID does not exist" Nov 01 12:48:14.559374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.559338 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm"] Nov 01 12:48:14.564069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.564042 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-83558e4e-openshift-default-7f84d78f87-q2srm"] Nov 01 12:48:14.811710 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.811680 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:14.991324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:14.990280 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:15.111427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.110030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:15.235925 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.235896 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656"] Nov 01 12:48:15.236594 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236543 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236606 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236619 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236625 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236637 4484 cpu_manager.go:414] "containerMap: removing container" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236643 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236655 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" Nov 01 12:48:15.236691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.236660 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" Nov 01 12:48:15.237574 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.237545 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" Nov 01 12:48:15.237684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.237587 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" Nov 01 12:48:15.237684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.237611 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" containerName="istio-proxy" Nov 01 12:48:15.237684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.237626 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc69f0c7-b732-4628-94ba-bf43ee87cee6" containerName="istio-proxy" Nov 01 12:48:15.246822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.246798 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.293011 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.292982 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656"] Nov 01 12:48:15.296056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.296029 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-61ffb38d\"" Nov 01 12:48:15.296200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.296181 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-61ffb38d-dockercfg-7v89d\"" Nov 01 12:48:15.315836 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.315804 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.316022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.315998 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnd67\" (UniqueName: \"kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67\") pod \"payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.451375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.419460 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.624325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.615104 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnd67\" (UniqueName: \"kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67\") pod \"payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.674325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.558100 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.674325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.659074 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj"] Nov 01 12:48:15.713251 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.713176 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj"] Nov 01 12:48:15.713501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.713251 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:15.719904 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.719882 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnd67\" (UniqueName: \"kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67\") pod \"payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.789822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.789797 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:15.840256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.840227 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lpj\" (UniqueName: \"kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj\") pod \"payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:15.840497 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.840475 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:15.862385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.862357 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:15.942924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.942497 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:15.942924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.942668 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lpj\" (UniqueName: \"kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj\") pod \"payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:15.943470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.943442 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:15.995741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:15.995710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lpj\" (UniqueName: \"kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj\") pod \"payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:16.041721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.041691 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:16.105688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.105015 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:16.253380 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.253349 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656"] Nov 01 12:48:16.256656 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:48:16.256613 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54cc796_66c4_4028_b762_de40352b02e3.slice/crio-e0b88c6caba417756a314b7d50deb2989e6b91fa3623bdc7373fd85052ff11cd WatchSource:0}: Error finding container e0b88c6caba417756a314b7d50deb2989e6b91fa3623bdc7373fd85052ff11cd: Status 404 returned error can't find the container with id e0b88c6caba417756a314b7d50deb2989e6b91fa3623bdc7373fd85052ff11cd Nov 01 12:48:16.371260 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.371232 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b768de-eab0-41ed-b2ee-4a7baf0122de" path="/var/lib/kubelet/pods/59b768de-eab0-41ed-b2ee-4a7baf0122de/volumes" Nov 01 12:48:16.417461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.417432 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj"] Nov 01 12:48:16.422803 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:48:16.422776 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc599ab_2dfe_4c81_aeca_a42a35c27ee3.slice/crio-8fab5a681e181bf417b5b0b0fb70f16200983354334b6d1ebbb0fe9ae4c00047 WatchSource:0}: Error finding container 8fab5a681e181bf417b5b0b0fb70f16200983354334b6d1ebbb0fe9ae4c00047: Status 404 returned error can't find the container with id 8fab5a681e181bf417b5b0b0fb70f16200983354334b6d1ebbb0fe9ae4c00047 Nov 01 12:48:16.434687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.434664 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:16.768324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.761627 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" event={"ID":"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3","Type":"ContainerStarted","Data":"a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9"} Nov 01 12:48:16.768324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.761692 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" event={"ID":"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3","Type":"ContainerStarted","Data":"8fab5a681e181bf417b5b0b0fb70f16200983354334b6d1ebbb0fe9ae4c00047"} Nov 01 12:48:16.768324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.761930 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:16.804325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.803037 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" event={"ID":"c54cc796-66c4-4028-b762-de40352b02e3","Type":"ContainerStarted","Data":"833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831"} Nov 01 12:48:16.804325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.803102 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" event={"ID":"c54cc796-66c4-4028-b762-de40352b02e3","Type":"ContainerStarted","Data":"e0b88c6caba417756a314b7d50deb2989e6b91fa3623bdc7373fd85052ff11cd"} Nov 01 12:48:16.804325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.804065 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:16.827455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.825610 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:16.881345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.881278 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" podStartSLOduration=1.881264362 podStartE2EDuration="1.881264362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:16.88063173 +0000 UTC m=+1566.630574709" watchObservedRunningTime="2026-11-01 12:48:16.881264362 +0000 UTC m=+1566.631207340" Nov 01 12:48:16.995358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:16.995294 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" podStartSLOduration=1.9952799369999998 podStartE2EDuration="1.995279937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:16.990466615 +0000 UTC m=+1566.740409593" watchObservedRunningTime="2026-11-01 12:48:16.995279937 +0000 UTC m=+1566.745222915" Nov 01 12:48:17.030752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:17.030682 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:17.162323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:17.160878 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:17.426523 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:17.426442 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:17.593284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:17.593252 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:19.992076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:19.992043 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:20.148848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.148812 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:20.362375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.359189 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:20.759424 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.758159 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl"] Nov 01 12:48:20.770589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.770556 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:20.840590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.840559 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e58e0934\"" Nov 01 12:48:20.841489 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.841468 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e58e0934-dockercfg-kb5hk\"" Nov 01 12:48:20.844708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.844653 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl"] Nov 01 12:48:20.872595 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.872557 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbqs\" (UniqueName: \"kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs\") pod \"payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:20.872984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.872960 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume\") pod \"payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:20.948730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:20.948698 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg"] Nov 01 12:48:21.034455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.023990 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume\") pod \"payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:21.035437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.035398 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbqs\" (UniqueName: \"kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs\") pod \"payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:21.118338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.101646 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume\") pod \"payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:21.137457 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.124996 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbqs\" (UniqueName: \"kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs\") pod \"payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:21.150136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.150105 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:21.163914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.163887 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg"] Nov 01 12:48:21.164414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.163963 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.224973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.224914 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:21.262550 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.262465 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhlc\" (UniqueName: \"kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc\") pod \"payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.262693 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.262631 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.369398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.368513 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhlc\" (UniqueName: \"kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc\") pod \"payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.369398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.368690 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.373330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.369716 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.423365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.423258 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhlc\" (UniqueName: \"kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc\") pod \"payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.513024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.512993 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:21.551295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.551259 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl"] Nov 01 12:48:21.848226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.848184 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" event={"ID":"d98b33cf-67f7-43d9-99af-6eeea8cecd75","Type":"ContainerStarted","Data":"f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d"} Nov 01 12:48:21.848478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.848257 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" event={"ID":"d98b33cf-67f7-43d9-99af-6eeea8cecd75","Type":"ContainerStarted","Data":"ea50945859365fca26d6e86b93accc7e275b8e675349fd8e0a5ed77e1b72f250"} Nov 01 12:48:21.848478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.848280 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:21.893230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.893153 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg"] Nov 01 12:48:21.899491 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:48:21.899453 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bc38ed_67cb_4600_b188_30302c0690bc.slice/crio-79765151695bb23265be9f027596bd5795aea0c99a484e66882b4aa1e69c9496 WatchSource:0}: Error finding container 79765151695bb23265be9f027596bd5795aea0c99a484e66882b4aa1e69c9496: Status 404 returned error can't find the container with id 79765151695bb23265be9f027596bd5795aea0c99a484e66882b4aa1e69c9496 Nov 01 12:48:21.956543 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:21.956497 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" podStartSLOduration=1.9564827729999998 podStartE2EDuration="1.956482773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:21.952960304 +0000 UTC m=+1571.702903281" watchObservedRunningTime="2026-11-01 12:48:21.956482773 +0000 UTC m=+1571.706425751" Nov 01 12:48:22.857029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:22.856992 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" event={"ID":"c2bc38ed-67cb-4600-b188-30302c0690bc","Type":"ContainerStarted","Data":"45a5f50a06b6dfb8ea7bce6198b63539c3ef105274b85da9832400703c051c1e"} Nov 01 12:48:22.857453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:22.857047 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" event={"ID":"c2bc38ed-67cb-4600-b188-30302c0690bc","Type":"ContainerStarted","Data":"79765151695bb23265be9f027596bd5795aea0c99a484e66882b4aa1e69c9496"} Nov 01 12:48:22.857453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:22.857068 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:22.965267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:22.965211 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" podStartSLOduration=2.965195433 podStartE2EDuration="2.965195433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:22.963255308 +0000 UTC m=+1572.713198287" watchObservedRunningTime="2026-11-01 12:48:22.965195433 +0000 UTC m=+1572.715138450" Nov 01 12:48:25.756563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:25.756489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:26.793617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:26.793578 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:27.813456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:27.813427 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:48:28.820555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:28.820521 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:48:30.252993 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:30.252954 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:48:32.120606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:32.120571 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:32.860630 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:32.860598 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:48:33.835763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:33.835727 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:48:33.868839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:33.868810 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:48:33.887656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:33.887625 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:37.394461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.394424 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9"] Nov 01 12:48:37.400163 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.400138 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.451549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.451525 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9"] Nov 01 12:48:37.493112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.493084 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-6c650b-serving-cert\"" Nov 01 12:48:37.528192 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.528167 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.528289 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.528235 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.528385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.528367 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzfl\" (UniqueName: \"kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.528518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.528499 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.613026 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.612997 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:37.630015 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.629987 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.630166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.630148 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.630240 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.630230 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.630336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.630313 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzfl\" (UniqueName: \"kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.630476 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:37.630291 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-serving-cert: secret "maas-api-e2e-shared-a-w1-6c650b-serving-cert" not found Nov 01 12:48:37.630545 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:48:37.630506 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls podName:1eb9367c-4699-4cba-af72-48ed1912e2be nodeName:}" failed. No retries permitted until 2026-11-01 12:48:38.130486433 +0000 UTC m=+1587.880429389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls") pod "maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" (UID: "1eb9367c-4699-4cba-af72-48ed1912e2be") : secret "maas-api-e2e-shared-a-w1-6c650b-serving-cert" not found Nov 01 12:48:37.630824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.630806 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.632616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.632595 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.734056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.734022 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzfl\" (UniqueName: \"kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:37.806348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:37.800710 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:38.033496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:38.033414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:38.145711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:38.145664 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:38.148106 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:38.148085 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:38.318874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:38.318805 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:38.695151 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:38.695074 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9"] Nov 01 12:48:38.700698 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:48:38.700672 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb9367c_4699_4cba_af72_48ed1912e2be.slice/crio-7310a421b5cbb1236c3f0167cc629dfcb08c1447fdd5168aa25b76ea914e71fb WatchSource:0}: Error finding container 7310a421b5cbb1236c3f0167cc629dfcb08c1447fdd5168aa25b76ea914e71fb: Status 404 returned error can't find the container with id 7310a421b5cbb1236c3f0167cc629dfcb08c1447fdd5168aa25b76ea914e71fb Nov 01 12:48:39.056804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:39.056764 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" event={"ID":"1eb9367c-4699-4cba-af72-48ed1912e2be","Type":"ContainerStarted","Data":"7310a421b5cbb1236c3f0167cc629dfcb08c1447fdd5168aa25b76ea914e71fb"} Nov 01 12:48:40.065170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:40.065076 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" event={"ID":"1eb9367c-4699-4cba-af72-48ed1912e2be","Type":"ContainerStarted","Data":"0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93"} Nov 01 12:48:40.065170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:40.065142 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:40.191527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:40.191454 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" podStartSLOduration=2.150224312 podStartE2EDuration="3.191441066s" totalImagesPullingTime="1.041216754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:40.188058483 +0000 UTC m=+1589.938001461" watchObservedRunningTime="2026-11-01 12:48:40.191441066 +0000 UTC m=+1589.941384088" Nov 01 12:48:40.606800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:40.606756 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:43.322459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:43.322431 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:46.090071 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:46.090043 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:48:48.702100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:48.702070 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:53.991432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:53.991383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:58.351876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:58.351846 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:59.031044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:59.031006 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:59.436231 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:59.436114 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:48:59.782709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:48:59.782675 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:00.251764 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:49:00.251721 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:49:03.474104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:03.474074 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:03.689620 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:03.689590 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:03.814694 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:03.814666 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:04.051460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:04.051429 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp"] Nov 01 12:49:04.097463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:04.097390 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f86820-f75fp"] Nov 01 12:49:04.371414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:04.371340 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38424d43-d66b-468b-89c5-ac839414c91d" path="/var/lib/kubelet/pods/38424d43-d66b-468b-89c5-ac839414c91d/volumes" Nov 01 12:49:04.869292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:04.869252 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:05.059826 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:05.059789 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:05.179008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:05.178874 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:08.034008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.033974 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:08.172560 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.172516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:08.459871 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.459735 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s"] Nov 01 12:49:08.468688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.465802 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.557376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.557274 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s"] Nov 01 12:49:08.566942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.566267 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.572317 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.572266 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpwh\" (UniqueName: \"kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh\") pod \"payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.616706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.616671 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-47feb945-dockercfg-jpbqm\"" Nov 01 12:49:08.623739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.623706 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-47feb945\"" Nov 01 12:49:08.710428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.689824 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.782880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.782834 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpwh\" (UniqueName: \"kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh\") pod \"payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.818323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.692248 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.942496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.942342 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:08.946886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.946860 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv"] Nov 01 12:49:08.967233 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.967146 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpwh\" (UniqueName: \"kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh\") pod \"payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:08.969153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.969129 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:08.972350 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:08.972326 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv"] Nov 01 12:49:09.069168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.069131 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtr8\" (UniqueName: \"kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8\") pod \"payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.069666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.069286 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume\") pod \"payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.132330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.132287 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:09.171577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.171530 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtr8\" (UniqueName: \"kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8\") pod \"payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.171817 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.171798 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume\") pod \"payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.173114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.173085 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume\") pod \"payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.225960 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.225920 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtr8\" (UniqueName: \"kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8\") pod \"payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.295831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.295799 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:09.510418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.510343 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s"] Nov 01 12:49:09.515047 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:49:09.515023 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b526f_80e9_48f2_91f9_396bc2d68a9d.slice/crio-13fd1881c8acd67e14cd0a86550e2c1559f451949e1a79fe31795ce5f7cf0f80 WatchSource:0}: Error finding container 13fd1881c8acd67e14cd0a86550e2c1559f451949e1a79fe31795ce5f7cf0f80: Status 404 returned error can't find the container with id 13fd1881c8acd67e14cd0a86550e2c1559f451949e1a79fe31795ce5f7cf0f80 Nov 01 12:49:09.673797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:09.673759 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv"] Nov 01 12:49:09.678612 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:49:09.678589 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fdad7c_9e56_4f31_a066_636bf30274c3.slice/crio-8f06bcc3e5bd680090e814737726fb59c92eaf4b73f1220bfa164be04d18e2c7 WatchSource:0}: Error finding container 8f06bcc3e5bd680090e814737726fb59c92eaf4b73f1220bfa164be04d18e2c7: Status 404 returned error can't find the container with id 8f06bcc3e5bd680090e814737726fb59c92eaf4b73f1220bfa164be04d18e2c7 Nov 01 12:49:10.052103 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.052073 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt"] Nov 01 12:49:10.056504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.056483 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.105610 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.105585 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e58e0934-serving-cert\"" Nov 01 12:49:10.110425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.110394 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt"] Nov 01 12:49:10.188266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.188233 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.188469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.188339 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.188512 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.188478 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.188561 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.188538 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb596\" (UniqueName: \"kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.290327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.290264 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.290517 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.290345 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb596\" (UniqueName: \"kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.290589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.290523 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.290646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.290611 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.290706 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:49:10.290670 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-serving-cert: secret "maas-api-e2e-ait-e58e0934-serving-cert" not found Nov 01 12:49:10.290760 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:49:10.290741 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls podName:cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8 nodeName:}" failed. No retries permitted until 2026-11-01 12:49:10.790720748 +0000 UTC m=+1620.540663725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls") pod "maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" (UID: "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8") : secret "maas-api-e2e-ait-e58e0934-serving-cert" not found Nov 01 12:49:10.291328 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.291288 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.292953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.292929 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.300833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.300810 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:10.346146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.346081 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb596\" (UniqueName: \"kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.415451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.415409 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" event={"ID":"9e6b526f-80e9-48f2-91f9-396bc2d68a9d","Type":"ContainerStarted","Data":"f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475"} Nov 01 12:49:10.415652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.415467 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" event={"ID":"9e6b526f-80e9-48f2-91f9-396bc2d68a9d","Type":"ContainerStarted","Data":"13fd1881c8acd67e14cd0a86550e2c1559f451949e1a79fe31795ce5f7cf0f80"} Nov 01 12:49:10.415652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.415495 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:10.417252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.417230 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" event={"ID":"f7fdad7c-9e56-4f31-a066-636bf30274c3","Type":"ContainerStarted","Data":"af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050"} Nov 01 12:49:10.417438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.417273 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:10.417438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.417290 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" event={"ID":"f7fdad7c-9e56-4f31-a066-636bf30274c3","Type":"ContainerStarted","Data":"8f06bcc3e5bd680090e814737726fb59c92eaf4b73f1220bfa164be04d18e2c7"} Nov 01 12:49:10.422327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.422284 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:10.541369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.541287 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" podStartSLOduration=2.541267626 podStartE2EDuration="2.541267626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:10.539190548 +0000 UTC m=+1620.289133529" watchObservedRunningTime="2026-11-01 12:49:10.541267626 +0000 UTC m=+1620.291210605" Nov 01 12:49:10.573734 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.542466 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:10.653250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.653127 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" podStartSLOduration=2.653110467 podStartE2EDuration="2.653110467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:10.648829653 +0000 UTC m=+1620.398772631" watchObservedRunningTime="2026-11-01 12:49:10.653110467 +0000 UTC m=+1620.403053470" Nov 01 12:49:10.708323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.705883 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:10.832321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.825616 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.966371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.948961 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\") pod \"maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:10.973786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:10.973749 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:11.330321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:11.326909 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:11.652970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:11.652944 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:49:11.710014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:11.709982 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:11.739849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:11.729687 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt"] Nov 01 12:49:12.440038 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:12.440003 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" event={"ID":"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8","Type":"ContainerStarted","Data":"748047c80a0a8de7d8fc3d96329cb18daba45d34de11d875f27844d4a6b1c645"} Nov 01 12:49:13.451377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:13.451338 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" event={"ID":"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8","Type":"ContainerStarted","Data":"c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb"} Nov 01 12:49:13.451825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:13.451423 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:13.569335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:13.569148 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" podStartSLOduration=2.653896161 podStartE2EDuration="3.569126139s" totalImagesPullingTime="915.229978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:13.564431578 +0000 UTC m=+1623.314374557" watchObservedRunningTime="2026-11-01 12:49:13.569126139 +0000 UTC m=+1623.319069117" Nov 01 12:49:16.220222 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:16.220187 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:16.576936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:16.576899 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:16.777318 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:16.777265 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:16.919940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:16.919853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:19.464338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:19.464276 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:49:20.356964 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:20.356934 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:20.529333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:20.526966 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:20.947178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:20.947140 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw"] Nov 01 12:49:20.954999 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:20.954837 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.001403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.001360 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw"] Nov 01 12:49:21.028322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.023489 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlt4j\" (UniqueName: \"kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j\") pod \"payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.028322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.023580 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.048337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.041661 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f6b043-dockercfg-xxgbl\"" Nov 01 12:49:21.048337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.041779 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f6b043\"" Nov 01 12:49:21.197372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.191477 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.239381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.202466 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.239381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.192663 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlt4j\" (UniqueName: \"kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j\") pod \"payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.385901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.385342 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlt4j\" (UniqueName: \"kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j\") pod \"payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.402034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.402005 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl"] Nov 01 12:49:21.407280 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.407260 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.407927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.407896 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl"] Nov 01 12:49:21.435692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.435664 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:49:21.436001 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.435981 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:49:21.484419 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.484382 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm927\" (UniqueName: \"kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927\") pod \"payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.484567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.484518 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.586500 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.586463 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.586928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.586715 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm927\" (UniqueName: \"kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927\") pod \"payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.587371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.587350 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.603157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.603136 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:21.641322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.641272 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm927\" (UniqueName: \"kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927\") pod \"payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.727576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.727537 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:21.999734 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:21.999652 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw"] Nov 01 12:49:22.004518 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:49:22.004490 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7725753_6369_44c9_bfdd_ed87b011d181.slice/crio-4128750bb4e4f536aea95e0f7661c7969a94fbfa28316042db818e921bf221bb WatchSource:0}: Error finding container 4128750bb4e4f536aea95e0f7661c7969a94fbfa28316042db818e921bf221bb: Status 404 returned error can't find the container with id 4128750bb4e4f536aea95e0f7661c7969a94fbfa28316042db818e921bf221bb Nov 01 12:49:22.132903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.132867 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl"] Nov 01 12:49:22.138052 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:49:22.138019 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fcb15f_20a0_49ec_a402_b983b05f66f0.slice/crio-881378186e8e8ae2883191144c242bc0723dc39c4748bedfdb097f6e1d266950 WatchSource:0}: Error finding container 881378186e8e8ae2883191144c242bc0723dc39c4748bedfdb097f6e1d266950: Status 404 returned error can't find the container with id 881378186e8e8ae2883191144c242bc0723dc39c4748bedfdb097f6e1d266950 Nov 01 12:49:22.280475 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.280397 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:22.657443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.657337 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" event={"ID":"93fcb15f-20a0-49ec-a402-b983b05f66f0","Type":"ContainerStarted","Data":"3f1dc7253b17e2e2d5b91a1a336027d2ed1d7ae85ae9352c83df15689587285e"} Nov 01 12:49:22.657443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.657406 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" event={"ID":"93fcb15f-20a0-49ec-a402-b983b05f66f0","Type":"ContainerStarted","Data":"881378186e8e8ae2883191144c242bc0723dc39c4748bedfdb097f6e1d266950"} Nov 01 12:49:22.657443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.657435 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:22.658990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.658962 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" event={"ID":"d7725753-6369-44c9-bfdd-ed87b011d181","Type":"ContainerStarted","Data":"28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d"} Nov 01 12:49:22.659130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.659006 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" event={"ID":"d7725753-6369-44c9-bfdd-ed87b011d181","Type":"ContainerStarted","Data":"4128750bb4e4f536aea95e0f7661c7969a94fbfa28316042db818e921bf221bb"} Nov 01 12:49:22.659130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.659027 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:22.766674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.766621 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" podStartSLOduration=1.7666054180000001 podStartE2EDuration="1.766605418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:22.76451919 +0000 UTC m=+1632.514462168" watchObservedRunningTime="2026-11-01 12:49:22.766605418 +0000 UTC m=+1632.516548396" Nov 01 12:49:22.874212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:22.874152 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" podStartSLOduration=2.874138016 podStartE2EDuration="2.874138016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:22.870577817 +0000 UTC m=+1632.620520796" watchObservedRunningTime="2026-11-01 12:49:22.874138016 +0000 UTC m=+1632.624080994" Nov 01 12:49:28.089152 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:28.089114 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:30.252883 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:49:30.252838 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:49:33.670208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:33.670177 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:49:33.670687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:33.670262 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:49:33.853358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:33.853314 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:39.044944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:39.044911 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j"] Nov 01 12:49:39.091101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:39.091069 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3440bb-5kj9j"] Nov 01 12:49:39.207852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:39.207821 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:40.371684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:40.371651 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec7e532-35e8-4450-8006-9494ab757de7" path="/var/lib/kubelet/pods/dec7e532-35e8-4450-8006-9494ab757de7/volumes" Nov 01 12:49:44.711992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:44.711961 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:45.985415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:45.985381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:47.360508 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:47.360476 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:47.780983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:47.780913 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:48.435955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:48.435918 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:48.901681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:48.894027 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:49.738319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:49.736815 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:49:50.213434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:50.213354 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:50.337852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:50.337816 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:50.466608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:50.466528 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:51.114325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:51.109331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:51.248076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:51.247494 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:54.302606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:54.302529 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:54.555389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:54.553888 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:54.681321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:54.680804 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:55.101190 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:55.101159 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk"] Nov 01 12:49:55.108136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:55.108091 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9cbc73-9gmhk"] Nov 01 12:49:56.397327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:56.396202 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee95598-ed34-425f-a0c1-f1ab511886fc" path="/var/lib/kubelet/pods/7ee95598-ed34-425f-a0c1-f1ab511886fc/volumes" Nov 01 12:49:56.933937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:56.933910 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:57.795521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:57.795426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:58.084378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.061875 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:49:58.605332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.603707 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv"] Nov 01 12:49:58.645429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.632367 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv"] Nov 01 12:49:58.645429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.632537 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:58.752739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.737260 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-0cd09f-dockercfg-9z8fk\"" Nov 01 12:49:58.752739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.737395 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-0cd09f\"" Nov 01 12:49:58.802713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.802671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnrh\" (UniqueName: \"kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh\") pod \"payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:58.803359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.803333 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:58.905683 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.905593 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnrh\" (UniqueName: \"kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh\") pod \"payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:58.905847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.905735 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:58.906523 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.906504 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:58.958470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:58.958442 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnrh\" (UniqueName: \"kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh\") pod \"payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:59.071938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:59.071909 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:49:59.434513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:59.434459 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:49:59.434690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:59.434550 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:49:59.455239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:49:59.455207 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv"] Nov 01 12:49:59.459435 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:49:59.459391 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85d3fb7_bd4a_41f8_8274_3efda26c7f03.slice/crio-8fd45624d9780a4e9f21391a7ff390d8d53c1f6815fc4a3dbae9a056f6c98c95 WatchSource:0}: Error finding container 8fd45624d9780a4e9f21391a7ff390d8d53c1f6815fc4a3dbae9a056f6c98c95: Status 404 returned error can't find the container with id 8fd45624d9780a4e9f21391a7ff390d8d53c1f6815fc4a3dbae9a056f6c98c95 Nov 01 12:50:00.257165 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:00.257129 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:50:00.270996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:00.270966 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" event={"ID":"b85d3fb7-bd4a-41f8-8274-3efda26c7f03","Type":"ContainerStarted","Data":"7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea"} Nov 01 12:50:00.271159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:00.271022 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" event={"ID":"b85d3fb7-bd4a-41f8-8274-3efda26c7f03","Type":"ContainerStarted","Data":"8fd45624d9780a4e9f21391a7ff390d8d53c1f6815fc4a3dbae9a056f6c98c95"} Nov 01 12:50:00.271159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:00.271053 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:50:00.471789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:00.471719 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" podStartSLOduration=2.471696296 podStartE2EDuration="2.471696296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:00.396316626 +0000 UTC m=+1670.146259607" watchObservedRunningTime="2026-11-01 12:50:00.471696296 +0000 UTC m=+1670.221639277" Nov 01 12:50:02.290903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.290830 4484 generic.go:356] "Generic (PLEG): container finished" podID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerID="b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4" exitCode=0 Nov 01 12:50:02.291363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.290907 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" event={"ID":"8a37a6db-f4b0-4ffe-b0be-1746671f12fa","Type":"ContainerDied","Data":"b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4"} Nov 01 12:50:02.291462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.291421 4484 scope.go:122] "RemoveContainer" containerID="b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4" Nov 01 12:50:02.292765 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.292742 4484 generic.go:356] "Generic (PLEG): container finished" podID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerID="2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c" exitCode=0 Nov 01 12:50:02.292867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.292811 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" event={"ID":"51727984-f64c-422b-ba4c-c9c1d3f6d414","Type":"ContainerDied","Data":"2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c"} Nov 01 12:50:02.293170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.293157 4484 scope.go:122] "RemoveContainer" containerID="2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c" Nov 01 12:50:02.293262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.293235 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:50:02.293341 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.293325 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:50:02.293382 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.293363 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:50:02.294734 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.294706 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:50:02.294813 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.294771 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:50:02.294861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.294810 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:50:02.580437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:02.580405 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:03.302217 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.302186 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" event={"ID":"8a37a6db-f4b0-4ffe-b0be-1746671f12fa","Type":"ContainerStarted","Data":"031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c"} Nov 01 12:50:03.304460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.304435 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" event={"ID":"51727984-f64c-422b-ba4c-c9c1d3f6d414","Type":"ContainerStarted","Data":"be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140"} Nov 01 12:50:03.669812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.669720 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:03.670119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.670089 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:03.670576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.669746 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:50:03.721714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.721673 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:50:03.721937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.721907 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 12:50:03.721985 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:03.721969 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 12:50:04.675323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:04.675035 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:04.675323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:04.675100 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:04.721907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:04.721869 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 12:50:04.722074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:04.721921 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 12:50:05.663755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:05.663717 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:05.664587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:05.664549 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:05.721690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:05.721659 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 12:50:05.722076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:05.721720 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 12:50:06.091616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.091426 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h"] Nov 01 12:50:06.097148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.097114 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-814980d2-cp87h"] Nov 01 12:50:06.396818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.396737 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf8256d-ade3-4ce8-be64-404de4d0fc46" path="/var/lib/kubelet/pods/6cf8256d-ade3-4ce8-be64-404de4d0fc46/volumes" Nov 01 12:50:06.670615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.670512 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:06.670959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.670923 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:06.721815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.721778 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 12:50:06.722199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:06.721839 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 12:50:07.674413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.674374 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:07.674709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.674679 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:07.698016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.697756 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc"] Nov 01 12:50:07.703792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.703769 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.722202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.722181 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 12:50:07.722484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.722227 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 12:50:07.777326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.773371 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-47feb945-serving-cert\"" Nov 01 12:50:07.813034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.813002 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc"] Nov 01 12:50:07.813271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.813122 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhv5\" (UniqueName: \"kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.813492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.813472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.813743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.813718 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.814678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.814655 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.828686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.828660 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:07.921521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.921469 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhv5\" (UniqueName: \"kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.921876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.921849 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.922244 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.922217 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.922410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.922382 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:07.922857 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:07.922826 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-serving-cert: secret "maas-api-e2e-mt-47feb945-serving-cert" not found Nov 01 12:50:07.923041 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:07.923018 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls podName:986a09ca-99b6-47cd-a9ba-c5c71fcb070f nodeName:}" failed. No retries permitted until 2026-11-01 12:50:08.422997983 +0000 UTC m=+1678.172940953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls") pod "maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" (UID: "986a09ca-99b6-47cd-a9ba-c5c71fcb070f") : secret "maas-api-e2e-mt-47feb945-serving-cert" not found Nov 01 12:50:07.923169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:07.923142 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:08.038328 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.034732 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:08.046336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.044905 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:08.115336 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.111097 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhv5\" (UniqueName: \"kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:08.422373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.417681 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:08.495587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.495534 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:08.502326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.498567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\") pod \"maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:08.566325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.564249 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:08.618945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.618913 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:08.688558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.681654 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:08.688558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.681712 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:08.721690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.721659 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 01 12:50:08.721849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:08.721706 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 01 12:50:09.142337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.138045 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:09.551329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.533811 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:09.578788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.578746 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc"] Nov 01 12:50:09.663259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.663024 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 12:50:09.663259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.663094 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 12:50:09.803812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.803664 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:50:09.804722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.804697 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:50:09.806212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.806166 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:50:09.993554 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:09.993523 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:10.564488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:10.564458 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" event={"ID":"986a09ca-99b6-47cd-a9ba-c5c71fcb070f","Type":"ContainerStarted","Data":"b66d31f7a57ee53ef6a305f40ca3dca8a4e239f0d6b79ec44a6def4f4a372290"} Nov 01 12:50:10.667029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:10.667002 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:50:10.667541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:10.667517 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:50:10.668253 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:10.668230 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:50:11.062482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:11.062445 4484 scope.go:122] "RemoveContainer" containerID="a40b34e5f2d07b04d5acbfa4c492c9b312baf8402912e0c459da1f4153428f3b" Nov 01 12:50:11.326373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:11.308892 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:50:11.575258 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:11.575215 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" event={"ID":"986a09ca-99b6-47cd-a9ba-c5c71fcb070f","Type":"ContainerStarted","Data":"c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358"} Nov 01 12:50:11.575773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:11.575679 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:11.683843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:11.683748 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" podStartSLOduration=3.824652378 podStartE2EDuration="4.683733191s" totalImagesPullingTime="859.080813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:11.682422527 +0000 UTC m=+1681.432365502" watchObservedRunningTime="2026-11-01 12:50:11.683733191 +0000 UTC m=+1681.433676222" Nov 01 12:50:13.716389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:13.716352 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:14.115781 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:14.115745 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:14.321920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:14.321889 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:14.465720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:14.465635 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:17.043961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:17.043927 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv"] Nov 01 12:50:17.090033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:17.089996 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befbc351-978zv"] Nov 01 12:50:17.549828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:17.549795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:17.608824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:17.608799 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:50:18.370874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:18.370841 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7658bfe-9033-4d52-960d-6086b78082a2" path="/var/lib/kubelet/pods/e7658bfe-9033-4d52-960d-6086b78082a2/volumes" Nov 01 12:50:18.996700 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:18.996667 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:19.230324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:19.207829 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:19.352403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:19.349503 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:23.137113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:23.137076 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:25.330996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:25.330965 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:25.588575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:25.588481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:25.702886 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:25.702857 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:30.253188 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:30.253149 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:50:31.283068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:31.283035 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:32.212106 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:32.212071 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:36.627218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:36.627186 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj"] Nov 01 12:50:36.731840 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:36.731805 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656"] Nov 01 12:50:36.784516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:36.784479 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:36.989514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:36.989408 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:37.106098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:37.106058 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:38.232324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:38.231129 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:50:40.791599 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:40.791565 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl"] Nov 01 12:50:40.907912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:40.907872 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg"] Nov 01 12:50:41.139067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.138993 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:41.342138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.341946 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:41.628049 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.628014 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" podUID="4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" containerName="payload-processing" containerID="cri-o://a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9" gracePeriod=25 Nov 01 12:50:41.732489 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.732435 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" podUID="c54cc796-66c4-4028-b762-de40352b02e3" containerName="payload-pre-processing" containerID="cri-o://833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831" gracePeriod=25 Nov 01 12:50:41.923816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.923788 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:50:41.956399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.956367 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj\" (UniqueName: \"kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj\") pod \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " Nov 01 12:50:41.956637 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.956536 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume\") pod \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\" (UID: \"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3\") " Nov 01 12:50:41.957702 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.956996 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume" pod "4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" (UID: "4fc599ab-2dfe-4c81-aeca-a42a35c27ee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:50:41.957702 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.957669 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:41.964389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:41.964199 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj" pod "4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" (UID: "4fc599ab-2dfe-4c81-aeca-a42a35c27ee3"). InnerVolumeSpecName "kube-api-access-w8lpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:50:42.071339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.066708 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lpj\" (UniqueName: \"kubernetes.io/projected/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3-kube-api-access-w8lpj\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:42.071339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.070731 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:50:42.097434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.097396 4484 generic.go:356] "Generic (PLEG): container finished" podID="c54cc796-66c4-4028-b762-de40352b02e3" containerID="833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831" exitCode=0 Nov 01 12:50:42.097633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.097456 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" Nov 01 12:50:42.097633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.097491 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" event={"ID":"c54cc796-66c4-4028-b762-de40352b02e3","Type":"ContainerDied","Data":"833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831"} Nov 01 12:50:42.097633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.097620 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656" event={"ID":"c54cc796-66c4-4028-b762-de40352b02e3","Type":"ContainerDied","Data":"e0b88c6caba417756a314b7d50deb2989e6b91fa3623bdc7373fd85052ff11cd"} Nov 01 12:50:42.097810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.097653 4484 scope.go:122] "RemoveContainer" containerID="833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831" Nov 01 12:50:42.099255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.099222 4484 generic.go:356] "Generic (PLEG): container finished" podID="4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" containerID="a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9" exitCode=0 Nov 01 12:50:42.099400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.099280 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" Nov 01 12:50:42.099400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.099337 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" event={"ID":"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3","Type":"ContainerDied","Data":"a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9"} Nov 01 12:50:42.099400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.099379 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj" event={"ID":"4fc599ab-2dfe-4c81-aeca-a42a35c27ee3","Type":"ContainerDied","Data":"8fab5a681e181bf417b5b0b0fb70f16200983354334b6d1ebbb0fe9ae4c00047"} Nov 01 12:50:42.110948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.110928 4484 scope.go:122] "RemoveContainer" containerID="833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831" Nov 01 12:50:42.111280 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:42.111261 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831\": container with ID starting with 833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831 not found: ID does not exist" containerID="833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831" Nov 01 12:50:42.111371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.111288 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831"} err="failed to get container status \"833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831\": rpc error: code = NotFound desc = could not find container \"833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831\": container with ID starting with 833f0cd8bff56fdfc86f0f0c812900003a18fb1c8b118207b899d46ce7041831 not found: ID does not exist" Nov 01 12:50:42.111371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.111320 4484 scope.go:122] "RemoveContainer" containerID="a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9" Nov 01 12:50:42.120739 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.120720 4484 scope.go:122] "RemoveContainer" containerID="a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9" Nov 01 12:50:42.121005 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:42.120988 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9\": container with ID starting with a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9 not found: ID does not exist" containerID="a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9" Nov 01 12:50:42.121069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.121010 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9"} err="failed to get container status \"a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9\": rpc error: code = NotFound desc = could not find container \"a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9\": container with ID starting with a6e5261f73fff467fede9fd10b38723b707cf7d92214cfde2e061cd56f445ca9 not found: ID does not exist" Nov 01 12:50:42.168193 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.168109 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume\") pod \"c54cc796-66c4-4028-b762-de40352b02e3\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " Nov 01 12:50:42.168381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.168340 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67\" (UniqueName: \"kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67\") pod \"c54cc796-66c4-4028-b762-de40352b02e3\" (UID: \"c54cc796-66c4-4028-b762-de40352b02e3\") " Nov 01 12:50:42.168549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.168520 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume" pod "c54cc796-66c4-4028-b762-de40352b02e3" (UID: "c54cc796-66c4-4028-b762-de40352b02e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:50:42.168997 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.168975 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54cc796-66c4-4028-b762-de40352b02e3-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:42.170403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.170383 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67" pod "c54cc796-66c4-4028-b762-de40352b02e3" (UID: "c54cc796-66c4-4028-b762-de40352b02e3"). InnerVolumeSpecName "kube-api-access-qnd67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:50:42.270551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.270520 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnd67\" (UniqueName: \"kubernetes.io/projected/c54cc796-66c4-4028-b762-de40352b02e3-kube-api-access-qnd67\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:42.281364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.281335 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj"] Nov 01 12:50:42.285483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.285448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:42.287218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.287198 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ffb38d-6f5556c9c7-kdqnj"] Nov 01 12:50:42.583373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.573587 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" path="/var/lib/kubelet/pods/4fc599ab-2dfe-4c81-aeca-a42a35c27ee3/volumes" Nov 01 12:50:42.666345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.662284 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656"] Nov 01 12:50:42.678330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:42.677163 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ffb38d-bd89756b7-fb656"] Nov 01 12:50:44.372005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:44.371969 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54cc796-66c4-4028-b762-de40352b02e3" path="/var/lib/kubelet/pods/c54cc796-66c4-4028-b762-de40352b02e3/volumes" Nov 01 12:50:45.792880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:45.792823 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" podUID="d98b33cf-67f7-43d9-99af-6eeea8cecd75" containerName="payload-processing" containerID="cri-o://f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d" gracePeriod=25 Nov 01 12:50:45.908552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:45.908501 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" podUID="c2bc38ed-67cb-4600-b188-30302c0690bc" containerName="payload-pre-processing" containerID="cri-o://45a5f50a06b6dfb8ea7bce6198b63539c3ef105274b85da9832400703c051c1e" gracePeriod=25 Nov 01 12:50:46.077553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.077523 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:50:46.107926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.107898 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs\" (UniqueName: \"kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs\") pod \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " Nov 01 12:50:46.108089 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.107936 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume\") pod \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\" (UID: \"d98b33cf-67f7-43d9-99af-6eeea8cecd75\") " Nov 01 12:50:46.108441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.108414 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume" pod "d98b33cf-67f7-43d9-99af-6eeea8cecd75" (UID: "d98b33cf-67f7-43d9-99af-6eeea8cecd75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:50:46.108795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.108774 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b33cf-67f7-43d9-99af-6eeea8cecd75-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:46.111131 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.111104 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs" pod "d98b33cf-67f7-43d9-99af-6eeea8cecd75" (UID: "d98b33cf-67f7-43d9-99af-6eeea8cecd75"). InnerVolumeSpecName "kube-api-access-dgbqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:50:46.143524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.143491 4484 generic.go:356] "Generic (PLEG): container finished" podID="c2bc38ed-67cb-4600-b188-30302c0690bc" containerID="45a5f50a06b6dfb8ea7bce6198b63539c3ef105274b85da9832400703c051c1e" exitCode=0 Nov 01 12:50:46.143709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.143690 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" event={"ID":"c2bc38ed-67cb-4600-b188-30302c0690bc","Type":"ContainerDied","Data":"45a5f50a06b6dfb8ea7bce6198b63539c3ef105274b85da9832400703c051c1e"} Nov 01 12:50:46.145478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.145453 4484 generic.go:356] "Generic (PLEG): container finished" podID="d98b33cf-67f7-43d9-99af-6eeea8cecd75" containerID="f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d" exitCode=0 Nov 01 12:50:46.145601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.145531 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" event={"ID":"d98b33cf-67f7-43d9-99af-6eeea8cecd75","Type":"ContainerDied","Data":"f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d"} Nov 01 12:50:46.145601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.145563 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" event={"ID":"d98b33cf-67f7-43d9-99af-6eeea8cecd75","Type":"ContainerDied","Data":"ea50945859365fca26d6e86b93accc7e275b8e675349fd8e0a5ed77e1b72f250"} Nov 01 12:50:46.145601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.145584 4484 scope.go:122] "RemoveContainer" containerID="f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d" Nov 01 12:50:46.145765 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.145699 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl" Nov 01 12:50:46.164587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.164563 4484 scope.go:122] "RemoveContainer" containerID="f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d" Nov 01 12:50:46.164858 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:50:46.164834 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d\": container with ID starting with f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d not found: ID does not exist" containerID="f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d" Nov 01 12:50:46.164956 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.164864 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d"} err="failed to get container status \"f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d\": rpc error: code = NotFound desc = could not find container \"f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d\": container with ID starting with f213b4a869d9e5527ed12e4257d0358d6699e64aeee29a3b71f080f2b581246d not found: ID does not exist" Nov 01 12:50:46.183013 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.182991 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:50:46.209561 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.209533 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc\" (UniqueName: \"kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc\") pod \"c2bc38ed-67cb-4600-b188-30302c0690bc\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " Nov 01 12:50:46.209713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.209681 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume\") pod \"c2bc38ed-67cb-4600-b188-30302c0690bc\" (UID: \"c2bc38ed-67cb-4600-b188-30302c0690bc\") " Nov 01 12:50:46.210111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.210070 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume" pod "c2bc38ed-67cb-4600-b188-30302c0690bc" (UID: "c2bc38ed-67cb-4600-b188-30302c0690bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:50:46.210505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.210486 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2bc38ed-67cb-4600-b188-30302c0690bc-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:46.210505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.210506 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbqs\" (UniqueName: \"kubernetes.io/projected/d98b33cf-67f7-43d9-99af-6eeea8cecd75-kube-api-access-dgbqs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:46.211685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.211663 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc" pod "c2bc38ed-67cb-4600-b188-30302c0690bc" (UID: "c2bc38ed-67cb-4600-b188-30302c0690bc"). InnerVolumeSpecName "kube-api-access-kbhlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:50:46.312177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.312099 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhlc\" (UniqueName: \"kubernetes.io/projected/c2bc38ed-67cb-4600-b188-30302c0690bc-kube-api-access-kbhlc\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:50:46.395226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.395197 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl"] Nov 01 12:50:46.401095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:46.401074 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e58e0934-68c77d5957-5tckl"] Nov 01 12:50:47.155516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:47.155427 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" event={"ID":"c2bc38ed-67cb-4600-b188-30302c0690bc","Type":"ContainerDied","Data":"79765151695bb23265be9f027596bd5795aea0c99a484e66882b4aa1e69c9496"} Nov 01 12:50:47.155516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:47.155505 4484 scope.go:122] "RemoveContainer" containerID="45a5f50a06b6dfb8ea7bce6198b63539c3ef105274b85da9832400703c051c1e" Nov 01 12:50:47.155997 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:47.155452 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg" Nov 01 12:50:47.308218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:47.308185 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg"] Nov 01 12:50:47.353894 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:47.353864 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e58e0934-846486c65f-jptlg"] Nov 01 12:50:47.626914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:47.626881 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:48.370887 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:48.370856 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bc38ed-67cb-4600-b188-30302c0690bc" path="/var/lib/kubelet/pods/c2bc38ed-67cb-4600-b188-30302c0690bc/volumes" Nov 01 12:50:48.371291 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:48.371275 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98b33cf-67f7-43d9-99af-6eeea8cecd75" path="/var/lib/kubelet/pods/d98b33cf-67f7-43d9-99af-6eeea8cecd75/volumes" Nov 01 12:50:52.875744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:52.875707 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:56.181917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:56.181884 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:50:58.179239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:50:58.179205 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:00.253512 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:00.253469 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:51:03.586376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:03.586341 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:04.704349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:04.704289 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l"] Nov 01 12:51:04.704861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:04.704577 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" containerID="cri-o://f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640" gracePeriod=30 Nov 01 12:51:04.747675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:04.747640 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68"] Nov 01 12:51:04.747912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:04.747887 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" containerID="cri-o://be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7" gracePeriod=30 Nov 01 12:51:04.753776 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:04.753749 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx"] Nov 01 12:51:04.753994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:04.753973 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" containerID="cri-o://a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b" gracePeriod=30 Nov 01 12:51:08.797273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:08.797240 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:08.951907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:08.951866 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:09.081376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:09.079425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:09.768079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:09.768037 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:10.082864 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.082612 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:51:10.098804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.098767 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.098875 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.098913 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.098968 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.099049 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.099083 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.099132 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.099186 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.099878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.099852 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.101470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.100165 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.101470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.100387 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.102613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.101685 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.102613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.101851 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5\") pod \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\" (UID: \"70a8fe3d-eea9-4247-a45b-b9d0f3da3255\") " Nov 01 12:51:10.104941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.104814 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.104941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.104845 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.104941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.104859 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.104941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.104874 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.105536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.105508 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:51:10.108632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.108553 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "kube-api-access-vsrx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:10.113656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.113632 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.115162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.115126 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:51:10.117827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.117804 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token" pod "70a8fe3d-eea9-4247-a45b-b9d0f3da3255" (UID: "70a8fe3d-eea9-4247-a45b-b9d0f3da3255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:10.143270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.143244 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:51:10.147278 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.147253 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:51:10.206109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206077 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.206285 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206146 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.206285 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206208 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.206285 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.206506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206291 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.206506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206380 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.206506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206475 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.206742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206506 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.206742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206534 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.206742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206588 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.206742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206731 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206796 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206826 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206845 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206828 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206866 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206929 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.206964 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.207030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207001 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket\") pod \"30058662-905d-4c70-958e-35729cf7a4ec\" (UID: \"30058662-905d-4c70-958e-35729cf7a4ec\") " Nov 01 12:51:10.207487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207081 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket\") pod \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\" (UID: \"bc76e96d-8c09-4f99-a9bd-0f526f7016a3\") " Nov 01 12:51:10.207487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207098 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207816 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207877 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207883 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30058662-905d-4c70-958e-35729cf7a4ec-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207943 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207960 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.207973 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208009 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208025 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrx5\" (UniqueName: \"kubernetes.io/projected/70a8fe3d-eea9-4247-a45b-b9d0f3da3255-kube-api-access-vsrx5\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208026 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208041 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208076 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208244 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208371 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208461 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:51:10.208503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.208466 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.225365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.213428 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:51:10.225365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.215584 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:51:10.225898 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.225866 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "kube-api-access-xfdgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:10.225999 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.225971 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.226449 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.226408 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:10.226632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.226611 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "kube-api-access-fwwxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:10.226867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.226841 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy" pod "30058662-905d-4c70-958e-35729cf7a4ec" (UID: "30058662-905d-4c70-958e-35729cf7a4ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:10.227950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.227926 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token" pod "bc76e96d-8c09-4f99-a9bd-0f526f7016a3" (UID: "bc76e96d-8c09-4f99-a9bd-0f526f7016a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:10.308975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308889 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30058662-905d-4c70-958e-35729cf7a4ec-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.308975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308920 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwxr\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-kube-api-access-fwwxr\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.308975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308938 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.308975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308951 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.308975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308964 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.308975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308977 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.308990 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309002 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfdgp\" (UniqueName: \"kubernetes.io/projected/30058662-905d-4c70-958e-35729cf7a4ec-kube-api-access-xfdgp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309014 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309026 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309038 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309050 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309062 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309074 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30058662-905d-4c70-958e-35729cf7a4ec-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.309319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.309086 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc76e96d-8c09-4f99-a9bd-0f526f7016a3-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:10.340046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.340016 4484 generic.go:356] "Generic (PLEG): container finished" podID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerID="a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b" exitCode=0 Nov 01 12:51:10.340167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.340051 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" event={"ID":"bc76e96d-8c09-4f99-a9bd-0f526f7016a3","Type":"ContainerDied","Data":"a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b"} Nov 01 12:51:10.340167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.340072 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" Nov 01 12:51:10.340167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.340110 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx" event={"ID":"bc76e96d-8c09-4f99-a9bd-0f526f7016a3","Type":"ContainerDied","Data":"dc842d9f606569433e85289d506d8bd696a3e247fe300913aa2d60c739c1a5cf"} Nov 01 12:51:10.340167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.340123 4484 scope.go:122] "RemoveContainer" containerID="a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b" Nov 01 12:51:10.341698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.341674 4484 generic.go:356] "Generic (PLEG): container finished" podID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerID="f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640" exitCode=0 Nov 01 12:51:10.341810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.341730 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" Nov 01 12:51:10.341810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.341752 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" event={"ID":"70a8fe3d-eea9-4247-a45b-b9d0f3da3255","Type":"ContainerDied","Data":"f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640"} Nov 01 12:51:10.341810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.341807 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l" event={"ID":"70a8fe3d-eea9-4247-a45b-b9d0f3da3255","Type":"ContainerDied","Data":"53c715b50fa624a0df4f40778ac50e1c869ac52b802595682fd870242b5747bc"} Nov 01 12:51:10.345590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.345561 4484 generic.go:356] "Generic (PLEG): container finished" podID="30058662-905d-4c70-958e-35729cf7a4ec" containerID="be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7" exitCode=0 Nov 01 12:51:10.345726 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.345636 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" event={"ID":"30058662-905d-4c70-958e-35729cf7a4ec","Type":"ContainerDied","Data":"be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7"} Nov 01 12:51:10.345726 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.345677 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" event={"ID":"30058662-905d-4c70-958e-35729cf7a4ec","Type":"ContainerDied","Data":"a27779c696056f6a4e1911e51cfd8735c0d411eec657dd3891eecd2bc401a6b6"} Nov 01 12:51:10.345839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.345760 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68" Nov 01 12:51:10.389722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.389702 4484 scope.go:122] "RemoveContainer" containerID="a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b" Nov 01 12:51:10.390689 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:10.390287 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b\": container with ID starting with a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b not found: ID does not exist" containerID="a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b" Nov 01 12:51:10.390850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.390821 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b"} err="failed to get container status \"a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b\": rpc error: code = NotFound desc = could not find container \"a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b\": container with ID starting with a796001a34baecb951473d4e451181d9b44959d160c981afb08f4819e673cf6b not found: ID does not exist" Nov 01 12:51:10.390958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.390945 4484 scope.go:122] "RemoveContainer" containerID="f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640" Nov 01 12:51:10.406073 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.406047 4484 scope.go:122] "RemoveContainer" containerID="f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640" Nov 01 12:51:10.406451 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:10.406426 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640\": container with ID starting with f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640 not found: ID does not exist" containerID="f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640" Nov 01 12:51:10.406514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.406461 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640"} err="failed to get container status \"f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640\": rpc error: code = NotFound desc = could not find container \"f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640\": container with ID starting with f7da69d4a68d3b006a81692dd26f1dc5ad6c0fa91e586ca61aeab36886abd640 not found: ID does not exist" Nov 01 12:51:10.406514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.406483 4484 scope.go:122] "RemoveContainer" containerID="be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7" Nov 01 12:51:10.418763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.418744 4484 scope.go:122] "RemoveContainer" containerID="be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7" Nov 01 12:51:10.418999 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:10.418980 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7\": container with ID starting with be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7 not found: ID does not exist" containerID="be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7" Nov 01 12:51:10.419076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.419008 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7"} err="failed to get container status \"be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7\": rpc error: code = NotFound desc = could not find container \"be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7\": container with ID starting with be7544b27d1b4a2af939fb16e1c6661ae702ce6927de7ad501a1bf7bc778e7a7 not found: ID does not exist" Nov 01 12:51:10.573565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.573489 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68"] Nov 01 12:51:10.577275 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.577251 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-8dl68"] Nov 01 12:51:10.671560 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.671517 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l"] Nov 01 12:51:10.678112 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.678086 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gdv4l"] Nov 01 12:51:10.781183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.781154 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx"] Nov 01 12:51:10.785280 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:10.785257 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-hstlx"] Nov 01 12:51:11.092880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:11.092848 4484 scope.go:122] "RemoveContainer" containerID="3fa86656988ff8ef57dac0c72f6a35a5238a07b487a87b57d8147addf227dab6" Nov 01 12:51:11.103758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:11.103733 4484 scope.go:122] "RemoveContainer" containerID="0a71d574af485402bb6b22b015ce94a0c4938d7741afd9e3d0546d129f8e481e" Nov 01 12:51:11.112704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:11.112679 4484 scope.go:122] "RemoveContainer" containerID="1a7fdfab7ba6dfc22f0545e3a68ba95f277035aef4419389c2252d7f4096e1f5" Nov 01 12:51:12.369911 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:12.369881 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30058662-905d-4c70-958e-35729cf7a4ec" path="/var/lib/kubelet/pods/30058662-905d-4c70-958e-35729cf7a4ec/volumes" Nov 01 12:51:12.370436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:12.370419 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" path="/var/lib/kubelet/pods/70a8fe3d-eea9-4247-a45b-b9d0f3da3255/volumes" Nov 01 12:51:12.370933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:12.370919 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" path="/var/lib/kubelet/pods/bc76e96d-8c09-4f99-a9bd-0f526f7016a3/volumes" Nov 01 12:51:14.354130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:14.354095 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:19.600239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:19.600199 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:24.645372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:24.637114 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:51:25.124054 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:25.124021 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:30.252482 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:30.252444 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:51:30.304935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.304907 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28"] Nov 01 12:51:30.305694 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305674 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d98b33cf-67f7-43d9-99af-6eeea8cecd75" containerName="payload-processing" Nov 01 12:51:30.305694 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305695 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98b33cf-67f7-43d9-99af-6eeea8cecd75" containerName="payload-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305710 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c54cc796-66c4-4028-b762-de40352b02e3" containerName="payload-pre-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305717 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54cc796-66c4-4028-b762-de40352b02e3" containerName="payload-pre-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305732 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c2bc38ed-67cb-4600-b188-30302c0690bc" containerName="payload-pre-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305739 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bc38ed-67cb-4600-b188-30302c0690bc" containerName="payload-pre-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305763 4484 cpu_manager.go:414] "containerMap: removing container" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305770 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305791 4484 cpu_manager.go:414] "containerMap: removing container" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305797 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305823 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" containerName="payload-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305827 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" containerName="payload-processing" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305835 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" Nov 01 12:51:30.305835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.305840 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306013 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bc38ed-67cb-4600-b188-30302c0690bc" containerName="payload-pre-processing" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306028 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a8fe3d-eea9-4247-a45b-b9d0f3da3255" containerName="istio-proxy" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306044 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="30058662-905d-4c70-958e-35729cf7a4ec" containerName="istio-proxy" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306063 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc599ab-2dfe-4c81-aeca-a42a35c27ee3" containerName="payload-processing" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306072 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc76e96d-8c09-4f99-a9bd-0f526f7016a3" containerName="istio-proxy" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306088 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98b33cf-67f7-43d9-99af-6eeea8cecd75" containerName="payload-processing" Nov 01 12:51:30.306230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.306097 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54cc796-66c4-4028-b762-de40352b02e3" containerName="payload-pre-processing" Nov 01 12:51:30.309906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.309885 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.373480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.373439 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28"] Nov 01 12:51:30.398859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.398824 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.399061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.399036 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.399197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.399178 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.399295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.399281 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdg28\" (UniqueName: \"kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.404075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.404055 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-187d06ed-serving-cert\"" Nov 01 12:51:30.500978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.500947 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.501134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.501047 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.501134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.501109 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdg28\" (UniqueName: \"kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.501611 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.501272 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.501611 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:30.501552 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-serving-cert: secret "maas-api-e2e-mt-187d06ed-serving-cert" not found Nov 01 12:51:30.501861 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:30.501690 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls podName:663731f6-87fc-4254-ac7f-b5469d6643f7 nodeName:}" failed. No retries permitted until 2026-11-01 12:51:31.001664348 +0000 UTC m=+1760.751607320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls") pod "maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" (UID: "663731f6-87fc-4254-ac7f-b5469d6643f7") : secret "maas-api-e2e-mt-187d06ed-serving-cert" not found Nov 01 12:51:30.501937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.501889 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.503991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.503941 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.507338 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.507309 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:30.554713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.554688 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdg28\" (UniqueName: \"kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:30.711551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.711515 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:30.849826 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:30.849742 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:31.014825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:31.014794 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:31.017341 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:31.017294 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:31.224558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:31.224464 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:31.343235 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:31.343195 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:31.599420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:31.599384 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28"] Nov 01 12:51:31.606827 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:51:31.606794 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663731f6_87fc_4254_ac7f_b5469d6643f7.slice/crio-1fb8777a8bcd0d13b8c44ea00c4612764b600811793f4bd9b7df6e42860e34f5 WatchSource:0}: Error finding container 1fb8777a8bcd0d13b8c44ea00c4612764b600811793f4bd9b7df6e42860e34f5: Status 404 returned error can't find the container with id 1fb8777a8bcd0d13b8c44ea00c4612764b600811793f4bd9b7df6e42860e34f5 Nov 01 12:51:31.648769 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:31.648740 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" event={"ID":"663731f6-87fc-4254-ac7f-b5469d6643f7","Type":"ContainerStarted","Data":"1fb8777a8bcd0d13b8c44ea00c4612764b600811793f4bd9b7df6e42860e34f5"} Nov 01 12:51:32.180199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:32.180163 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:32.315404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:32.315371 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:32.657574 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:32.657536 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" event={"ID":"663731f6-87fc-4254-ac7f-b5469d6643f7","Type":"ContainerStarted","Data":"88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed"} Nov 01 12:51:32.657937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:32.657670 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:32.767543 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:32.767494 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" podStartSLOduration=1.904319556 podStartE2EDuration="2.767479223s" totalImagesPullingTime="863.159667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:32.765877357 +0000 UTC m=+1762.515820332" watchObservedRunningTime="2026-11-01 12:51:32.767479223 +0000 UTC m=+1762.517422200" Nov 01 12:51:32.795917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:32.795883 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:35.829207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:35.829178 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:37.052286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.052251 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt"] Nov 01 12:51:37.052814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.052450 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" podUID="cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" containerName="maas-api" containerID="cri-o://c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb" gracePeriod=30 Nov 01 12:51:37.150259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.150141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:37.433862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.433585 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:51:37.575208 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.575172 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:37.607286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.607259 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\") pod \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " Nov 01 12:51:37.607458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.607368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs\") pod \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " Nov 01 12:51:37.607531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.607513 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca\") pod \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " Nov 01 12:51:37.607709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.607632 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596\" (UniqueName: \"kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596\") pod \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\" (UID: \"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8\") " Nov 01 12:51:37.607950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.607912 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca" pod "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" (UID: "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:51:37.608576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.608547 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:37.609691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.609664 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls" pod "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" (UID: "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:51:37.609940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.609919 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs" pod "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" (UID: "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:51:37.610119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.610097 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596" pod "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" (UID: "cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8"). InnerVolumeSpecName "kube-api-access-kb596". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:37.728814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.727936 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb596\" (UniqueName: \"kubernetes.io/projected/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-kube-api-access-kb596\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:37.728814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.727978 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:37.728814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.727992 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:37.769474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.760376 4484 generic.go:356] "Generic (PLEG): container finished" podID="cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" containerID="c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb" exitCode=0 Nov 01 12:51:37.769474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.760671 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" event={"ID":"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8","Type":"ContainerDied","Data":"c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb"} Nov 01 12:51:37.769474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.760727 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" event={"ID":"cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8","Type":"ContainerDied","Data":"748047c80a0a8de7d8fc3d96329cb18daba45d34de11d875f27844d4a6b1c645"} Nov 01 12:51:37.769474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.760756 4484 scope.go:122] "RemoveContainer" containerID="c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb" Nov 01 12:51:37.769474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.760918 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt" Nov 01 12:51:37.841526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.839188 4484 scope.go:122] "RemoveContainer" containerID="c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb" Nov 01 12:51:37.842033 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:37.841995 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb\": container with ID starting with c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb not found: ID does not exist" containerID="c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb" Nov 01 12:51:37.842215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.842188 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb"} err="failed to get container status \"c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb\": rpc error: code = NotFound desc = could not find container \"c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb\": container with ID starting with c9b4738740d344d309adc274679d4518b1519cd9b274bebae542e2a790bf1cdb not found: ID does not exist" Nov 01 12:51:37.927623 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.927596 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt"] Nov 01 12:51:37.929743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.929717 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:37.932132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:37.932110 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e58e0934-f66db5ccb-t9txt"] Nov 01 12:51:38.322688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:38.322652 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:38.541161 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:38.541122 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" path="/var/lib/kubelet/pods/cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8/volumes" Nov 01 12:51:38.542181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:38.542156 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh"] Nov 01 12:51:38.542715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:38.542688 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" containerID="cri-o://031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c" gracePeriod=30 Nov 01 12:51:38.742555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:38.742505 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:51:38.989442 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:38.989405 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:39.209570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.209462 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:39.326771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.326741 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:39.675788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.675759 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:39.821323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.821051 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:39.822220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.821745 4484 patch_prober.go:29] interesting pod/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:51:39.822220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.821806 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:51:39.950376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:39.945789 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:40.510541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:40.510452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:40.525172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:40.525148 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd"] Nov 01 12:51:40.525400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:40.525376 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" containerID="cri-o://be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140" gracePeriod=30 Nov 01 12:51:40.686353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:40.683584 4484 patch_prober.go:29] interesting pod/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:51:40.686353 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:40.683639 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:51:41.845737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:41.845700 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:41.985737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:41.985701 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:43.885397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.885371 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:51:43.900473 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.900444 4484 generic.go:356] "Generic (PLEG): container finished" podID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerID="031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c" exitCode=0 Nov 01 12:51:43.900592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.900503 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" Nov 01 12:51:43.900592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.900536 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" event={"ID":"8a37a6db-f4b0-4ffe-b0be-1746671f12fa","Type":"ContainerDied","Data":"031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c"} Nov 01 12:51:43.900666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.900591 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh" event={"ID":"8a37a6db-f4b0-4ffe-b0be-1746671f12fa","Type":"ContainerDied","Data":"03ba53250f395e77363bed042d18fa97c2150fee89f01bef2290562af8464444"} Nov 01 12:51:43.900666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.900624 4484 scope.go:122] "RemoveContainer" containerID="031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c" Nov 01 12:51:43.910530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.910512 4484 scope.go:122] "RemoveContainer" containerID="b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4" Nov 01 12:51:43.920348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.920288 4484 scope.go:122] "RemoveContainer" containerID="031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c" Nov 01 12:51:43.920628 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:43.920610 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c\": container with ID starting with 031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c not found: ID does not exist" containerID="031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c" Nov 01 12:51:43.920708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.920635 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c"} err="failed to get container status \"031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c\": rpc error: code = NotFound desc = could not find container \"031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c\": container with ID starting with 031ec38b99729db166cfce234c5e1775d5e741a18fa3cc02547917ae868e3f2c not found: ID does not exist" Nov 01 12:51:43.920708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.920655 4484 scope.go:122] "RemoveContainer" containerID="b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4" Nov 01 12:51:43.920901 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:43.920881 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4\": container with ID starting with b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4 not found: ID does not exist" containerID="b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4" Nov 01 12:51:43.920937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:43.920908 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4"} err="failed to get container status \"b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4\": rpc error: code = NotFound desc = could not find container \"b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4\": container with ID starting with b2425fd524b850b657fa9b5032608c128515655141be26fc4f49dd89053aa5d4 not found: ID does not exist" Nov 01 12:51:44.041659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.041622 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.041851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.041727 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.041851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.041777 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.041969 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.041925 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.042031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.041981 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.042031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.041997 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:44.042141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042058 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.042141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042091 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:51:44.042250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042182 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.042250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042229 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.042376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042241 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:44.042376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042252 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:44.042480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042395 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket\") pod \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\" (UID: \"8a37a6db-f4b0-4ffe-b0be-1746671f12fa\") " Nov 01 12:51:44.043181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.042693 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:44.043181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.043028 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.043181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.043049 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.043181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.043065 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.043181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.043081 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.043181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.043094 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.045002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.044968 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:51:44.045152 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.045130 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:44.045254 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.045223 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "kube-api-access-k875d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:44.045668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.045646 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy" pod "8a37a6db-f4b0-4ffe-b0be-1746671f12fa" (UID: "8a37a6db-f4b0-4ffe-b0be-1746671f12fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:44.145027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.144949 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.145027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.144980 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.145027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.144992 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k875d\" (UniqueName: \"kubernetes.io/projected/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-kube-api-access-k875d\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.145027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.145002 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a37a6db-f4b0-4ffe-b0be-1746671f12fa-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:44.319567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.319526 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh"] Nov 01 12:51:44.325863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.325834 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-61ffb38d-gw-openshift-default-844dcfcf5c-s62fh"] Nov 01 12:51:44.371396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:44.371363 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" path="/var/lib/kubelet/pods/8a37a6db-f4b0-4ffe-b0be-1746671f12fa/volumes" Nov 01 12:51:45.596522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.596485 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:45.806641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.806570 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:51:45.807214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.807190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:45.872751 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.872718 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.872916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.872792 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.872916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.872855 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.872931 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873005 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873183 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873249 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873328 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873399 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token\") pod \"51727984-f64c-422b-ba4c-c9c1d3f6d414\" (UID: \"51727984-f64c-422b-ba4c-c9c1d3f6d414\") " Nov 01 12:51:45.873492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873461 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:51:45.873634 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873610 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:45.873707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873636 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:45.873923 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873877 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:45.874044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.873943 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:45.874288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.874267 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.874417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.874404 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.874498 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.874486 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.874584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.874573 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51727984-f64c-422b-ba4c-c9c1d3f6d414-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.874663 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.874652 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.876244 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.876219 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:51:45.876389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.876351 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:51:45.876459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.876411 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "kube-api-access-brrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:45.876586 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.876563 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token" pod "51727984-f64c-422b-ba4c-c9c1d3f6d414" (UID: "51727984-f64c-422b-ba4c-c9c1d3f6d414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:51:45.917761 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.917732 4484 generic.go:356] "Generic (PLEG): container finished" podID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerID="be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140" exitCode=0 Nov 01 12:51:45.917879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.917780 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" Nov 01 12:51:45.917879 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.917817 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" event={"ID":"51727984-f64c-422b-ba4c-c9c1d3f6d414","Type":"ContainerDied","Data":"be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140"} Nov 01 12:51:45.917961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.917906 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd" event={"ID":"51727984-f64c-422b-ba4c-c9c1d3f6d414","Type":"ContainerDied","Data":"ba6036eb2f22c8ef005135a38701186ab6e31444e5d91770370b8b58abad3a7e"} Nov 01 12:51:45.917961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.917934 4484 scope.go:122] "RemoveContainer" containerID="be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140" Nov 01 12:51:45.929059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.929040 4484 scope.go:122] "RemoveContainer" containerID="2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c" Nov 01 12:51:45.929288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.929265 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:45.938571 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.938553 4484 scope.go:122] "RemoveContainer" containerID="be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140" Nov 01 12:51:45.938856 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:45.938836 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140\": container with ID starting with be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140 not found: ID does not exist" containerID="be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140" Nov 01 12:51:45.938920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.938865 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140"} err="failed to get container status \"be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140\": rpc error: code = NotFound desc = could not find container \"be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140\": container with ID starting with be28348a29a6cf336b1fc68a9cb10d06382fc4c97bb2d59fcd121863caf35140 not found: ID does not exist" Nov 01 12:51:45.938920 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.938882 4484 scope.go:122] "RemoveContainer" containerID="2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c" Nov 01 12:51:45.939114 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:51:45.939098 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c\": container with ID starting with 2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c not found: ID does not exist" containerID="2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c" Nov 01 12:51:45.939162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.939119 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c"} err="failed to get container status \"2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c\": rpc error: code = NotFound desc = could not find container \"2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c\": container with ID starting with 2e985910a32af9c08026c6f1d3f4abfa8fa8f672dfd6dec3f5181f09449c278c not found: ID does not exist" Nov 01 12:51:45.975940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.975909 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.975940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.975933 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.975940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.975945 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrzt\" (UniqueName: \"kubernetes.io/projected/51727984-f64c-422b-ba4c-c9c1d3f6d414-kube-api-access-brrzt\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:45.976150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:45.975955 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51727984-f64c-422b-ba4c-c9c1d3f6d414-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:51:46.072541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:46.072465 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd"] Nov 01 12:51:46.080553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:46.080528 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e58e0934-openshift-default-774bdb7859-p4rrd"] Nov 01 12:51:46.147229 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:46.147194 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:46.370264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:46.370186 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" path="/var/lib/kubelet/pods/51727984-f64c-422b-ba4c-c9c1d3f6d414/volumes" Nov 01 12:51:47.992895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:47.992867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:48.493847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.493816 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc"] Nov 01 12:51:48.494494 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494479 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" containerName="maas-api" Nov 01 12:51:48.494536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494495 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" containerName="maas-api" Nov 01 12:51:48.494536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494528 4484 cpu_manager.go:414] "containerMap: removing container" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494536 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494553 4484 cpu_manager.go:414] "containerMap: removing container" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494560 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494577 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494585 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494596 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" Nov 01 12:51:48.494606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494602 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" Nov 01 12:51:48.494815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494726 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" Nov 01 12:51:48.494815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494734 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="51727984-f64c-422b-ba4c-c9c1d3f6d414" containerName="istio-proxy" Nov 01 12:51:48.494815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494742 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" Nov 01 12:51:48.494815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494749 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a37a6db-f4b0-4ffe-b0be-1746671f12fa" containerName="istio-proxy" Nov 01 12:51:48.494815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.494760 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfedec4f-6a6b-4e18-9d10-a9a26e4cdaa8" containerName="maas-api" Nov 01 12:51:48.499793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.499775 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.560325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.558651 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-94c13a-openshift-default-dockercfg-2zqxw\"" Nov 01 12:51:48.589253 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.589161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:48.675420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.675382 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc"] Nov 01 12:51:48.732812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.725330 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswkh\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.733122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.733100 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.733332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.733294 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.759592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.759499 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.759852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.759830 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.759986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.759969 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.767365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.760264 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.767590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.767565 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.767758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.767741 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.768042 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.732596 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:48.885732 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.885692 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswkh\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.886087 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.886062 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.886331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.886311 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.886521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.886500 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.886708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.886690 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.886874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.886858 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.887164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.887140 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.887411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.887391 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.887590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.887571 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.888744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.888710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.889028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.888890 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.890071 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.889902 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.905368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.891210 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.905368 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.895625 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.951095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.949620 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.968465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.953080 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswkh\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.968465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.959780 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:48.994323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:48.993970 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo\") pod \"e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:49.113859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.113765 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:49.114565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.114530 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:49.256093 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.256046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:49.475259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.475190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:49.576354 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.576319 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc"] Nov 01 12:51:49.581799 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:51:49.581772 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a57d05_e15f_4aed_b79f_3245af6376a9.slice/crio-9ba73cf31b4237d1eac460183915f8aad9fc8c3c818616a87310bed36830fb00 WatchSource:0}: Error finding container 9ba73cf31b4237d1eac460183915f8aad9fc8c3c818616a87310bed36830fb00: Status 404 returned error can't find the container with id 9ba73cf31b4237d1eac460183915f8aad9fc8c3c818616a87310bed36830fb00 Nov 01 12:51:49.584615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.584578 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:51:49.584677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.584650 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:51:49.584713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.584679 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:51:49.653944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:49.653912 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:50.125323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:50.114532 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" event={"ID":"10a57d05-e15f-4aed-b79f-3245af6376a9","Type":"ContainerStarted","Data":"662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1"} Nov 01 12:51:50.125323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:50.114608 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" event={"ID":"10a57d05-e15f-4aed-b79f-3245af6376a9","Type":"ContainerStarted","Data":"9ba73cf31b4237d1eac460183915f8aad9fc8c3c818616a87310bed36830fb00"} Nov 01 12:51:50.317023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:50.316043 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podStartSLOduration=2.3160229709999998 podStartE2EDuration="2.316022971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:50.282119561 +0000 UTC m=+1780.032062577" watchObservedRunningTime="2026-11-01 12:51:50.316022971 +0000 UTC m=+1780.065965954" Nov 01 12:51:50.552273 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:50.552235 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:50.844714 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:50.844606 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:51.114397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:51.114287 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:51.120029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:51.120005 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:51.122107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:51.122091 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:51.123204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:51.123185 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:51:51.966860 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:51.966818 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:52.759722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:52.759693 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:53.167215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:53.167133 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:53.316144 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:53.316116 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:56.031352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:56.031291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:56.225894 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:56.225855 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:56.366167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:56.366046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:56.884203 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:56.884167 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:57.092937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:57.092896 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:51:58.110429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:51:58.110383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:00.253443 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:52:00.253408 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:52:01.762146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:01.762115 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:02.025394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:02.025315 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:03.690044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:03.690011 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:08.997697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:08.997658 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:09.924635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:09.924602 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:52:10.279006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:10.278974 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:10.395279 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:10.395253 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:11.181888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:11.181852 4484 scope.go:122] "RemoveContainer" containerID="2f5a6a2c538138ca8e483f1d9db232df8892b3ac676f0bcae620f2d2229048ad" Nov 01 12:52:14.411071 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:14.411040 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:14.546451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:14.546412 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:19.805333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:19.805272 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:20.275620 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:20.275586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:25.188033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:25.188000 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:30.253068 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:52:30.253021 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:52:30.699269 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:30.698009 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:32.663143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:32.663111 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:36.192831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:36.192785 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:41.506107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:41.506069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:46.156578 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:46.156526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:46.682692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:46.682657 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8"] Nov 01 12:52:46.689060 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:46.689038 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:46.831438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:46.824793 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:46.860323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:46.855655 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f893c2-openshift-default-dockercfg-rzs22\"" Nov 01 12:52:47.034039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:46.993580 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8"] Nov 01 12:52:47.034039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.020726 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:47.038293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.038254 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.038508 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.038413 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.038508 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.038459 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.038792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.038769 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4tq\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.038882 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.038867 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.038970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.038935 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.062370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.039032 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.062370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.047120 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.062370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.047162 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.156345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.151436 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.156345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.151563 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.156345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.151621 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.156345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.151649 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.156345 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.151703 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.164377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.157853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.196087 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.196158 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.196258 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4tq\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.196391 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.197378 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.197624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.204331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.202945 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:47.208567 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.208535 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.220456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.220410 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.235456 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.235418 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.257821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.251194 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.279322 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.273859 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.297228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.297164 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4tq\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq\") pod \"e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.306692 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.306666 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:47.738454 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.738425 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8"] Nov 01 12:52:47.744255 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:52:47.744225 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaae4295_9113_41d7_a6db_1d030f4590e5.slice/crio-4b993e904f565b9f8ff57d2842cf9c155ed5dbce8f05c04fbd15d6fc17e17c59 WatchSource:0}: Error finding container 4b993e904f565b9f8ff57d2842cf9c155ed5dbce8f05c04fbd15d6fc17e17c59: Status 404 returned error can't find the container with id 4b993e904f565b9f8ff57d2842cf9c155ed5dbce8f05c04fbd15d6fc17e17c59 Nov 01 12:52:47.746874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.746841 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:52:47.746963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.746906 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:52:47.746963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:47.746937 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:52:48.639974 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:48.639932 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" event={"ID":"baae4295-9113-41d7-a6db-1d030f4590e5","Type":"ContainerStarted","Data":"78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82"} Nov 01 12:52:48.640427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:48.640001 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" event={"ID":"baae4295-9113-41d7-a6db-1d030f4590e5","Type":"ContainerStarted","Data":"4b993e904f565b9f8ff57d2842cf9c155ed5dbce8f05c04fbd15d6fc17e17c59"} Nov 01 12:52:48.796647 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:48.796589 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podStartSLOduration=2.796574545 podStartE2EDuration="2.796574545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:48.7937406 +0000 UTC m=+1838.543683579" watchObservedRunningTime="2026-11-01 12:52:48.796574545 +0000 UTC m=+1838.546517522" Nov 01 12:52:49.306814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:49.306786 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:49.312450 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:49.312425 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:49.646174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:49.646098 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:49.647129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:49.647110 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:52:52.566476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:52.566439 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:52.995407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:52.995375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:53.300174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:53.300099 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:53.497090 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:53.497017 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:56.819508 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:56.819476 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:56.944849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:56.944818 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:57.350581 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:57.350551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:58.117091 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:58.117057 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:58.327412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:58.326406 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:58.489142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:58.489106 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:58.641464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:58.641430 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:52:58.827657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:52:58.814790 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:00.252667 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:00.252633 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:53:01.486505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:01.486477 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:04.426484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:04.426450 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:10.249573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:10.249530 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:10.475254 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:10.475215 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:10.603506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:10.600890 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:15.891926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:15.891894 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:17.412638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:17.412605 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv"] Nov 01 12:53:17.518601 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:17.518566 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s"] Nov 01 12:53:17.762154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:17.762124 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:17.995944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:17.995911 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:21.105631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:21.105600 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:21.288263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:21.288096 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:21.405218 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:21.405031 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:22.413534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.413479 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" podUID="f7fdad7c-9e56-4f31-a066-636bf30274c3" containerName="payload-processing" containerID="cri-o://af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050" gracePeriod=25 Nov 01 12:53:22.519333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.519262 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" podUID="9e6b526f-80e9-48f2-91f9-396bc2d68a9d" containerName="payload-pre-processing" containerID="cri-o://f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475" gracePeriod=25 Nov 01 12:53:22.732648 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.732619 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:53:22.774870 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.774840 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8\" (UniqueName: \"kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8\") pod \"f7fdad7c-9e56-4f31-a066-636bf30274c3\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " Nov 01 12:53:22.775099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.774945 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume\") pod \"f7fdad7c-9e56-4f31-a066-636bf30274c3\" (UID: \"f7fdad7c-9e56-4f31-a066-636bf30274c3\") " Nov 01 12:53:22.775419 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.775389 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume" pod "f7fdad7c-9e56-4f31-a066-636bf30274c3" (UID: "f7fdad7c-9e56-4f31-a066-636bf30274c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:22.776055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.776019 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7fdad7c-9e56-4f31-a066-636bf30274c3-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:22.778056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.778033 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8" pod "f7fdad7c-9e56-4f31-a066-636bf30274c3" (UID: "f7fdad7c-9e56-4f31-a066-636bf30274c3"). InnerVolumeSpecName "kube-api-access-wvtr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:22.850233 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.850193 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:53:22.876584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.876554 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh\" (UniqueName: \"kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh\") pod \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " Nov 01 12:53:22.876905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.876878 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume\") pod \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\" (UID: \"9e6b526f-80e9-48f2-91f9-396bc2d68a9d\") " Nov 01 12:53:22.877312 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.877274 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume" pod "9e6b526f-80e9-48f2-91f9-396bc2d68a9d" (UID: "9e6b526f-80e9-48f2-91f9-396bc2d68a9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:22.877619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.877600 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvtr8\" (UniqueName: \"kubernetes.io/projected/f7fdad7c-9e56-4f31-a066-636bf30274c3-kube-api-access-wvtr8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:22.877663 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.877625 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:22.878687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.878663 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh" pod "9e6b526f-80e9-48f2-91f9-396bc2d68a9d" (UID: "9e6b526f-80e9-48f2-91f9-396bc2d68a9d"). InnerVolumeSpecName "kube-api-access-phpwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:22.979250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:22.979201 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phpwh\" (UniqueName: \"kubernetes.io/projected/9e6b526f-80e9-48f2-91f9-396bc2d68a9d-kube-api-access-phpwh\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:23.143935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.143900 4484 generic.go:356] "Generic (PLEG): container finished" podID="f7fdad7c-9e56-4f31-a066-636bf30274c3" containerID="af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050" exitCode=0 Nov 01 12:53:23.144122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.143942 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" Nov 01 12:53:23.144122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.143988 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" event={"ID":"f7fdad7c-9e56-4f31-a066-636bf30274c3","Type":"ContainerDied","Data":"af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050"} Nov 01 12:53:23.144122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.144049 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv" event={"ID":"f7fdad7c-9e56-4f31-a066-636bf30274c3","Type":"ContainerDied","Data":"8f06bcc3e5bd680090e814737726fb59c92eaf4b73f1220bfa164be04d18e2c7"} Nov 01 12:53:23.144122 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.144077 4484 scope.go:122] "RemoveContainer" containerID="af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050" Nov 01 12:53:23.145449 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.145427 4484 generic.go:356] "Generic (PLEG): container finished" podID="9e6b526f-80e9-48f2-91f9-396bc2d68a9d" containerID="f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475" exitCode=0 Nov 01 12:53:23.145573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.145554 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" event={"ID":"9e6b526f-80e9-48f2-91f9-396bc2d68a9d","Type":"ContainerDied","Data":"f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475"} Nov 01 12:53:23.145612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.145602 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" event={"ID":"9e6b526f-80e9-48f2-91f9-396bc2d68a9d","Type":"ContainerDied","Data":"13fd1881c8acd67e14cd0a86550e2c1559f451949e1a79fe31795ce5f7cf0f80"} Nov 01 12:53:23.145685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.145673 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s" Nov 01 12:53:23.154710 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.154690 4484 scope.go:122] "RemoveContainer" containerID="af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050" Nov 01 12:53:23.154987 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:23.154968 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050\": container with ID starting with af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050 not found: ID does not exist" containerID="af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050" Nov 01 12:53:23.155061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.154996 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050"} err="failed to get container status \"af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050\": rpc error: code = NotFound desc = could not find container \"af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050\": container with ID starting with af80632d435d70373843e0876bd59d2742db0be9ac88f829e8754fe1694f7050 not found: ID does not exist" Nov 01 12:53:23.155061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.155014 4484 scope.go:122] "RemoveContainer" containerID="f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475" Nov 01 12:53:23.164009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.163994 4484 scope.go:122] "RemoveContainer" containerID="f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475" Nov 01 12:53:23.164275 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:23.164257 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475\": container with ID starting with f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475 not found: ID does not exist" containerID="f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475" Nov 01 12:53:23.164362 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.164284 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475"} err="failed to get container status \"f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475\": rpc error: code = NotFound desc = could not find container \"f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475\": container with ID starting with f172ab51cd1ad4012f771af934e544ceaf2a57d7046acddde727cd66fd91a475 not found: ID does not exist" Nov 01 12:53:23.318998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.318918 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s"] Nov 01 12:53:23.322639 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.322614 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47feb945-86758bcff9-2v42s"] Nov 01 12:53:23.422050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.422017 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv"] Nov 01 12:53:23.426545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:23.426523 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47feb945-7f5964b96c-f64dv"] Nov 01 12:53:24.398391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:24.398353 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6b526f-80e9-48f2-91f9-396bc2d68a9d" path="/var/lib/kubelet/pods/9e6b526f-80e9-48f2-91f9-396bc2d68a9d/volumes" Nov 01 12:53:24.399078 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:24.399057 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fdad7c-9e56-4f31-a066-636bf30274c3" path="/var/lib/kubelet/pods/f7fdad7c-9e56-4f31-a066-636bf30274c3/volumes" Nov 01 12:53:26.627660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:26.627631 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:30.253130 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:30.253092 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:53:31.955064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:31.955031 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:33.162253 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:33.162173 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:53:35.416796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:35.416763 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv"] Nov 01 12:53:35.417238 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:35.416971 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" containerID="cri-o://4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7" gracePeriod=30 Nov 01 12:53:35.672192 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:35.672098 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97"] Nov 01 12:53:35.672398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:35.672377 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" containerID="cri-o://d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b" gracePeriod=30 Nov 01 12:53:35.678459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:35.678431 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8"] Nov 01 12:53:35.678671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:35.678649 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" containerID="cri-o://c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1" gracePeriod=30 Nov 01 12:53:37.693922 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:37.693876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:37.865332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:37.862376 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4"] Nov 01 12:53:37.865332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:37.862656 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" containerID="cri-o://0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5" gracePeriod=30 Nov 01 12:53:37.920833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:37.919831 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh"] Nov 01 12:53:37.920833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:37.920580 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" containerID="cri-o://cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f" gracePeriod=30 Nov 01 12:53:38.687846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:38.687814 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:53:38.688025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:38.687865 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:53:39.035560 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:39.035516 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:53:39.035926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:39.035595 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:53:39.895809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:39.895778 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:53:39.895968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:39.895831 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:53:40.824524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.824495 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:53:40.858810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.858774 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.858994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.858823 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.858994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.858921 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.858994 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.858970 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.859169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859034 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.859169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859111 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.859285 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859237 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.859285 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859277 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.859401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859351 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert\") pod \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\" (UID: \"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8\") " Nov 01 12:53:40.859809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859620 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:40.860172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.859988 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:40.860172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860006 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:40.860403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860196 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:40.860403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860280 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:40.860403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860391 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.860545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860412 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.860545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860427 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.860545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860440 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.860545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.860454 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.862467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.862420 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "kube-api-access-m2kzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:40.863558 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.863375 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:40.880396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.864069 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:40.880396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.875404 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo" pod "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" (UID: "4a304bf5-a631-4fe9-b47c-6df19ef8b9d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:53:40.962327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.962231 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.962327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.962271 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.962327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.962289 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.962327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.962320 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2kzr\" (UniqueName: \"kubernetes.io/projected/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8-kube-api-access-m2kzr\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:40.978902 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:40.978870 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:53:41.030845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.030813 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:53:41.063256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063221 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063266 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063331 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.063455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063382 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063420 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063492 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063539 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.063697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063550 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.063697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063618 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.063906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063700 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063736 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.063906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063805 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.063906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063870 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.064181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063915 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.064181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063923 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:41.064181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.063969 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.064181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064080 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket\") pod \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\" (UID: \"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4\") " Nov 01 12:53:41.064181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064128 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.064181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064171 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.064522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064207 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert\") pod \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\" (UID: \"6b2d570e-41b7-429b-9d41-9ffc4e47dd73\") " Nov 01 12:53:41.064522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064215 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.064634 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064559 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.064704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064678 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.064799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064773 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.064905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064885 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.064996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.064974 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.065209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065183 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065212 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065229 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065245 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065259 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065273 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065285 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065331 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.065729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.065538 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.067287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.066562 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:41.068348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.068277 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:41.069222 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.068994 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.070469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.070416 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:53:41.070469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.070440 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "kube-api-access-cqcp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:41.070681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.070474 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:41.070681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.070506 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "kube-api-access-mnchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:41.074688 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.074657 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo" pod "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" (UID: "28af2f2e-7971-45d7-aa4d-bf8e5afc89b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:53:41.074766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.074716 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy" pod "6b2d570e-41b7-429b-9d41-9ffc4e47dd73" (UID: "6b2d570e-41b7-429b-9d41-9ffc4e47dd73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:41.167815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167764 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.167815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167816 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqcp2\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-kube-api-access-cqcp2\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167832 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167848 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167862 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167875 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167887 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167901 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167915 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2d570e-41b7-429b-9d41-9ffc4e47dd73-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.168064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.167929 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnchs\" (UniqueName: \"kubernetes.io/projected/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4-kube-api-access-mnchs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:41.334562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.334527 4484 generic.go:356] "Generic (PLEG): container finished" podID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerID="d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b" exitCode=0 Nov 01 12:53:41.334764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.334594 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" Nov 01 12:53:41.334764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.334610 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" event={"ID":"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4","Type":"ContainerDied","Data":"d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b"} Nov 01 12:53:41.334764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.334709 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97" event={"ID":"28af2f2e-7971-45d7-aa4d-bf8e5afc89b4","Type":"ContainerDied","Data":"f6d49241b8bafa72f9d72bd2c6976d3c5055c77a53829c6d78aed5d672cfa0e4"} Nov 01 12:53:41.334764 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.334748 4484 scope.go:122] "RemoveContainer" containerID="d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b" Nov 01 12:53:41.336437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.336413 4484 generic.go:356] "Generic (PLEG): container finished" podID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerID="c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1" exitCode=0 Nov 01 12:53:41.336624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.336608 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" Nov 01 12:53:41.336696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.336638 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" event={"ID":"6b2d570e-41b7-429b-9d41-9ffc4e47dd73","Type":"ContainerDied","Data":"c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1"} Nov 01 12:53:41.336696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.336678 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8" event={"ID":"6b2d570e-41b7-429b-9d41-9ffc4e47dd73","Type":"ContainerDied","Data":"dd9cbfd8bd6bad54e50e5f92f7e737790b46a0b15ea351be54890f63a2c152f4"} Nov 01 12:53:41.338448 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.338416 4484 generic.go:356] "Generic (PLEG): container finished" podID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerID="4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7" exitCode=0 Nov 01 12:53:41.338596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.338496 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" Nov 01 12:53:41.338596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.338542 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" event={"ID":"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8","Type":"ContainerDied","Data":"4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7"} Nov 01 12:53:41.338596 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.338581 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv" event={"ID":"4a304bf5-a631-4fe9-b47c-6df19ef8b9d8","Type":"ContainerDied","Data":"74ee3841da1596559f67820437ab29f1dd85d8d5a44dfe5bf28031a861f5f667"} Nov 01 12:53:41.364361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.364041 4484 scope.go:122] "RemoveContainer" containerID="d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b" Nov 01 12:53:41.365937 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:41.365410 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b\": container with ID starting with d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b not found: ID does not exist" containerID="d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b" Nov 01 12:53:41.365937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.365448 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b"} err="failed to get container status \"d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b\": rpc error: code = NotFound desc = could not find container \"d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b\": container with ID starting with d3510cd58cf26f3733221336c58fa12ef5894c6b20b4c52841f92499c3c8544b not found: ID does not exist" Nov 01 12:53:41.365937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.365476 4484 scope.go:122] "RemoveContainer" containerID="c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1" Nov 01 12:53:41.378893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.378870 4484 scope.go:122] "RemoveContainer" containerID="c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1" Nov 01 12:53:41.379458 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:41.379430 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1\": container with ID starting with c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1 not found: ID does not exist" containerID="c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1" Nov 01 12:53:41.379562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.379470 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1"} err="failed to get container status \"c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1\": rpc error: code = NotFound desc = could not find container \"c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1\": container with ID starting with c87e2dd8b34084a068c030a10a7664dfc3b2220e9b38f3454d9cbf4566115ee1 not found: ID does not exist" Nov 01 12:53:41.379562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.379498 4484 scope.go:122] "RemoveContainer" containerID="4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7" Nov 01 12:53:41.392814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.392797 4484 scope.go:122] "RemoveContainer" containerID="4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7" Nov 01 12:53:41.393125 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:41.393103 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7\": container with ID starting with 4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7 not found: ID does not exist" containerID="4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7" Nov 01 12:53:41.393204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.393130 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7"} err="failed to get container status \"4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7\": rpc error: code = NotFound desc = could not find container \"4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7\": container with ID starting with 4c3b158bd6c6d1426a424c95ca30b44e255e0100262efa950deb5bbf9f50b7b7 not found: ID does not exist" Nov 01 12:53:41.490027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.489994 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97"] Nov 01 12:53:41.498642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.498605 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qsv97"] Nov 01 12:53:41.606059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.605984 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv"] Nov 01 12:53:41.613625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.613597 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-bwjcv"] Nov 01 12:53:41.718431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.718389 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8"] Nov 01 12:53:41.728703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:41.728671 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mwfq8"] Nov 01 12:53:42.371098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:42.371064 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" path="/var/lib/kubelet/pods/28af2f2e-7971-45d7-aa4d-bf8e5afc89b4/volumes" Nov 01 12:53:42.371797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:42.371775 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" path="/var/lib/kubelet/pods/4a304bf5-a631-4fe9-b47c-6df19ef8b9d8/volumes" Nov 01 12:53:42.372276 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:42.372256 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" path="/var/lib/kubelet/pods/6b2d570e-41b7-429b-9d41-9ffc4e47dd73/volumes" Nov 01 12:53:43.082881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.082848 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:43.332114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.332080 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:53:43.341981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.341954 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:53:43.356681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.356652 4484 generic.go:356] "Generic (PLEG): container finished" podID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerID="cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f" exitCode=0 Nov 01 12:53:43.356831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.356722 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" Nov 01 12:53:43.356831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.356739 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" event={"ID":"0d6e017e-35a9-4f5c-a735-fc801396a2bb","Type":"ContainerDied","Data":"cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f"} Nov 01 12:53:43.356831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.356813 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh" event={"ID":"0d6e017e-35a9-4f5c-a735-fc801396a2bb","Type":"ContainerDied","Data":"dac2b23424ba6673b0da0f90b137f6cfbc3728128b598554fd4d4661a5907baa"} Nov 01 12:53:43.356982 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.356836 4484 scope.go:122] "RemoveContainer" containerID="cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f" Nov 01 12:53:43.358740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.358714 4484 generic.go:356] "Generic (PLEG): container finished" podID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerID="0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5" exitCode=0 Nov 01 12:53:43.358861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.358755 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" event={"ID":"78342bb9-80b3-4763-afdd-210e793e2bbb","Type":"ContainerDied","Data":"0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5"} Nov 01 12:53:43.358861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.358769 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" Nov 01 12:53:43.358861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.358788 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4" event={"ID":"78342bb9-80b3-4763-afdd-210e793e2bbb","Type":"ContainerDied","Data":"f3b4f4e495810a1382121a4d608c212f8a6548d6400c45fe9e556d01cd5e3368"} Nov 01 12:53:43.368849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.368824 4484 scope.go:122] "RemoveContainer" containerID="cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f" Nov 01 12:53:43.369127 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:43.369109 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f\": container with ID starting with cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f not found: ID does not exist" containerID="cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f" Nov 01 12:53:43.369195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.369133 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f"} err="failed to get container status \"cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f\": rpc error: code = NotFound desc = could not find container \"cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f\": container with ID starting with cf962dad26b4dfb467f438b72456669fc4fbdb16c158672fd5a960530096f00f not found: ID does not exist" Nov 01 12:53:43.369195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.369149 4484 scope.go:122] "RemoveContainer" containerID="0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5" Nov 01 12:53:43.378440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.378422 4484 scope.go:122] "RemoveContainer" containerID="fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646" Nov 01 12:53:43.387358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.387337 4484 scope.go:122] "RemoveContainer" containerID="0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5" Nov 01 12:53:43.387659 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:43.387638 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5\": container with ID starting with 0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5 not found: ID does not exist" containerID="0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5" Nov 01 12:53:43.387731 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.387668 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5"} err="failed to get container status \"0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5\": rpc error: code = NotFound desc = could not find container \"0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5\": container with ID starting with 0744a2460c3fb0be7fe09344477bbc2029f1d23e1b63664217e84aea4ca07cf5 not found: ID does not exist" Nov 01 12:53:43.387731 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.387685 4484 scope.go:122] "RemoveContainer" containerID="fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646" Nov 01 12:53:43.387944 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:53:43.387925 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646\": container with ID starting with fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646 not found: ID does not exist" containerID="fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646" Nov 01 12:53:43.387987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.387948 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646"} err="failed to get container status \"fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646\": rpc error: code = NotFound desc = could not find container \"fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646\": container with ID starting with fc26b3047e49fabd6f6253b2ccb42117b73df5e15271a05c2be1506b76805646 not found: ID does not exist" Nov 01 12:53:43.402608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402587 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.402689 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402628 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.402689 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402661 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.402795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402704 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.402795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402738 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.402795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402768 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.402957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402805 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.402957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402848 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.402957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402879 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.402957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402899 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.402957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402934 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.403216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.402978 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.403216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403072 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.403216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.403216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403116 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.403461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403272 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.403461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403318 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.403461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403329 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.403461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403338 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw\") pod \"78342bb9-80b3-4763-afdd-210e793e2bbb\" (UID: \"78342bb9-80b3-4763-afdd-210e793e2bbb\") " Nov 01 12:53:43.403461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403384 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.403461 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403433 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.403788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403467 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.403788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403514 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data\") pod \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\" (UID: \"0d6e017e-35a9-4f5c-a735-fc801396a2bb\") " Nov 01 12:53:43.403788 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.403639 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.404035 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404010 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404481 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404508 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404525 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404542 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404559 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78342bb9-80b3-4763-afdd-210e793e2bbb-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404574 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.404589 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.405928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.405812 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.405928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.405817 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.405928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.405870 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:53:43.406150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.406095 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "kube-api-access-jmdtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:43.406211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.406146 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:43.406211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.406176 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:43.406349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.406223 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.406349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.406292 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:53:43.406980 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.406958 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:53:43.407518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.407497 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo" pod "0d6e017e-35a9-4f5c-a735-fc801396a2bb" (UID: "0d6e017e-35a9-4f5c-a735-fc801396a2bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:53:43.407725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.407710 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw" pod "78342bb9-80b3-4763-afdd-210e793e2bbb" (UID: "78342bb9-80b3-4763-afdd-210e793e2bbb"). InnerVolumeSpecName "kube-api-access-qn2cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:53:43.506493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506460 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506493 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506506 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506518 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506531 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdtb\" (UniqueName: \"kubernetes.io/projected/0d6e017e-35a9-4f5c-a735-fc801396a2bb-kube-api-access-jmdtb\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506544 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78342bb9-80b3-4763-afdd-210e793e2bbb-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506555 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn2cw\" (UniqueName: \"kubernetes.io/projected/78342bb9-80b3-4763-afdd-210e793e2bbb-kube-api-access-qn2cw\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506568 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506580 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506592 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.506716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.506604 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6e017e-35a9-4f5c-a735-fc801396a2bb-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:53:43.802691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.802583 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh"] Nov 01 12:53:43.806251 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.806224 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-fz2fh"] Nov 01 12:53:43.911600 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.911562 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4"] Nov 01 12:53:43.918503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:43.918477 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k92f4"] Nov 01 12:53:44.369314 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.369263 4484 generic.go:356] "Generic (PLEG): container finished" podID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerID="662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1" exitCode=0 Nov 01 12:53:44.371655 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.371628 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" path="/var/lib/kubelet/pods/0d6e017e-35a9-4f5c-a735-fc801396a2bb/volumes" Nov 01 12:53:44.372132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.372119 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" path="/var/lib/kubelet/pods/78342bb9-80b3-4763-afdd-210e793e2bbb/volumes" Nov 01 12:53:44.372454 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.372437 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" event={"ID":"10a57d05-e15f-4aed-b79f-3245af6376a9","Type":"ContainerDied","Data":"662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1"} Nov 01 12:53:44.372820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.372806 4484 scope.go:122] "RemoveContainer" containerID="662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1" Nov 01 12:53:44.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.374230 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:53:44.374347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.374327 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:53:44.374399 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.374364 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:53:44.929095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:44.929053 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:45.407320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.407012 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" event={"ID":"10a57d05-e15f-4aed-b79f-3245af6376a9","Type":"ContainerStarted","Data":"7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813"} Nov 01 12:53:45.513698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.513666 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f"] Nov 01 12:53:45.514311 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514288 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9e6b526f-80e9-48f2-91f9-396bc2d68a9d" containerName="payload-pre-processing" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514313 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6b526f-80e9-48f2-91f9-396bc2d68a9d" containerName="payload-pre-processing" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514331 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f7fdad7c-9e56-4f31-a066-636bf30274c3" containerName="payload-processing" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514336 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fdad7c-9e56-4f31-a066-636bf30274c3" containerName="payload-processing" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514346 4484 cpu_manager.go:414] "containerMap: removing container" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514350 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514357 4484 cpu_manager.go:414] "containerMap: removing container" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514362 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" Nov 01 12:53:45.514376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514372 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514380 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514416 4484 cpu_manager.go:414] "containerMap: removing container" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514421 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514431 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514435 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514535 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6e017e-35a9-4f5c-a735-fc801396a2bb" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514543 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2d570e-41b7-429b-9d41-9ffc4e47dd73" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514551 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6b526f-80e9-48f2-91f9-396bc2d68a9d" containerName="payload-pre-processing" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514560 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fdad7c-9e56-4f31-a066-636bf30274c3" containerName="payload-processing" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514566 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514573 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="28af2f2e-7971-45d7-aa4d-bf8e5afc89b4" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514579 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a304bf5-a631-4fe9-b47c-6df19ef8b9d8" containerName="istio-proxy" Nov 01 12:53:45.514615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.514585 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" Nov 01 12:53:45.517016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.516992 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.560321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.542101 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f"] Nov 01 12:53:45.574924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.561773 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.575261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.575233 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.575526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.575508 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.575723 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.575701 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.575977 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.575958 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.576207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.576189 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.576632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.576565 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6b9030-openshift-default-dockercfg-gvwhs\"" Nov 01 12:53:45.576762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.576740 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.576865 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.576817 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjlk\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.576916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.576886 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.577905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.577879 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:45.678420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.678326 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjlk\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.678420 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.678403 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.678670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.678501 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.678670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.678544 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.678670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.678565 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679065 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679023 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679091 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679087 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679241 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679356 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679622 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679597 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679747 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.679973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.679953 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.681100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.681080 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.681573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.681556 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.733784 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.733757 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.734518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.734498 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjlk\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk\") pod \"e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:45.778323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.772093 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:45.879358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:45.877617 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:46.114173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.114113 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:53:46.114804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.114775 4484 patch_prober.go:29] interesting pod/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 12:53:46.114909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.114834 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 12:53:46.569436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.569407 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f"] Nov 01 12:53:46.571475 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:53:46.571449 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11835b7f_717d_48fa_8971_65b79dfa49d9.slice/crio-770df299a50d8acc1d59c44f431d516a880f80ef3b26f4ca896dd6a809f7185a WatchSource:0}: Error finding container 770df299a50d8acc1d59c44f431d516a880f80ef3b26f4ca896dd6a809f7185a: Status 404 returned error can't find the container with id 770df299a50d8acc1d59c44f431d516a880f80ef3b26f4ca896dd6a809f7185a Nov 01 12:53:46.574346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.574291 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:53:46.574458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.574392 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:53:46.574458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:46.574443 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:53:47.114656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.114620 4484 patch_prober.go:29] interesting pod/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 12:53:47.115055 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.114673 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 12:53:47.421632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.421541 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" event={"ID":"11835b7f-717d-48fa-8971-65b79dfa49d9","Type":"ContainerStarted","Data":"0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107"} Nov 01 12:53:47.421632 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.421601 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" event={"ID":"11835b7f-717d-48fa-8971-65b79dfa49d9","Type":"ContainerStarted","Data":"770df299a50d8acc1d59c44f431d516a880f80ef3b26f4ca896dd6a809f7185a"} Nov 01 12:53:47.533715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.533661 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podStartSLOduration=2.5336472150000002 podStartE2EDuration="2.533647215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:47.529762078 +0000 UTC m=+1897.279705056" watchObservedRunningTime="2026-11-01 12:53:47.533647215 +0000 UTC m=+1897.283590192" Nov 01 12:53:47.878163 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.878136 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:47.884176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:47.884150 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:48.114457 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:48.114420 4484 patch_prober.go:29] interesting pod/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 12:53:48.114613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:48.114475 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 12:53:48.428147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:48.428115 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:48.429204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:48.429185 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:53:48.711398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:48.711286 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:49.114607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:49.114570 4484 patch_prober.go:29] interesting pod/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 12:53:49.114869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:49.114621 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 12:53:50.114537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:50.114504 4484 patch_prober.go:29] interesting pod/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 12:53:50.115027 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:50.114558 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 12:53:50.408593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:50.408496 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:50.586061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:50.586030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:50.789333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:50.789276 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:51.118407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:51.118335 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:53:51.118804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:51.118623 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:53:51.119198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:51.119183 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 12:53:53.725189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:53.725158 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:53.953863 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:53.953825 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:54.177194 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.177162 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh"] Nov 01 12:53:54.179188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.179159 4484 cpu_manager.go:414] "containerMap: removing container" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" Nov 01 12:53:54.179188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.179187 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78342bb9-80b3-4763-afdd-210e793e2bbb" containerName="istio-proxy" Nov 01 12:53:54.186151 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.186130 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.204024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.203996 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh"] Nov 01 12:53:54.350059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.347337 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjj8\" (UniqueName: \"kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8\") pod \"payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.350059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.347588 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.350059 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.349511 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6b9030\"" Nov 01 12:53:54.350467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.350170 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6b9030-dockercfg-wthh9\"" Nov 01 12:53:54.357635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.357613 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5"] Nov 01 12:53:54.463364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.454560 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5"] Nov 01 12:53:54.463364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.454668 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.463364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.454790 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjj8\" (UniqueName: \"kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8\") pod \"payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.463364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.456102 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.463364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.456215 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.508676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.508641 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjj8\" (UniqueName: \"kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8\") pod \"payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:54.556010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.555965 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpqg\" (UniqueName: \"kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg\") pod \"payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.556259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.556238 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.608496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.608461 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:53:54.657401 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.657371 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.657577 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.657486 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpqg\" (UniqueName: \"kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg\") pod \"payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.658129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.658104 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.734232 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.734197 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpqg\" (UniqueName: \"kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg\") pod \"payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.790489 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.790458 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:54.805243 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:54.805220 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:55.176961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.176882 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5"] Nov 01 12:53:55.183286 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:53:55.183260 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237c99e9_2d6b_43e4_8858_8361d1fcc4f6.slice/crio-9fa333b79d622f7d99883272e00b6c32c9e646e1154c071852e4534cc18ad613 WatchSource:0}: Error finding container 9fa333b79d622f7d99883272e00b6c32c9e646e1154c071852e4534cc18ad613: Status 404 returned error can't find the container with id 9fa333b79d622f7d99883272e00b6c32c9e646e1154c071852e4534cc18ad613 Nov 01 12:53:55.229359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.229328 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh"] Nov 01 12:53:55.241590 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:53:55.241560 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fbf03b_f3e9_4994_9248_8f30e963748a.slice/crio-7cc53bdc2cbf7b1eb7ea2048b125c52eb0701a8d5ee04a4f145ac4de12447ce1 WatchSource:0}: Error finding container 7cc53bdc2cbf7b1eb7ea2048b125c52eb0701a8d5ee04a4f145ac4de12447ce1: Status 404 returned error can't find the container with id 7cc53bdc2cbf7b1eb7ea2048b125c52eb0701a8d5ee04a4f145ac4de12447ce1 Nov 01 12:53:55.511155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.511124 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" event={"ID":"56fbf03b-f3e9-4994-9248-8f30e963748a","Type":"ContainerStarted","Data":"b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249"} Nov 01 12:53:55.511363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.511179 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" event={"ID":"56fbf03b-f3e9-4994-9248-8f30e963748a","Type":"ContainerStarted","Data":"7cc53bdc2cbf7b1eb7ea2048b125c52eb0701a8d5ee04a4f145ac4de12447ce1"} Nov 01 12:53:55.511363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.511204 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:53:55.513140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.513116 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" event={"ID":"237c99e9-2d6b-43e4-8858-8361d1fcc4f6","Type":"ContainerStarted","Data":"7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b"} Nov 01 12:53:55.513261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.513154 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" event={"ID":"237c99e9-2d6b-43e4-8858-8361d1fcc4f6","Type":"ContainerStarted","Data":"9fa333b79d622f7d99883272e00b6c32c9e646e1154c071852e4534cc18ad613"} Nov 01 12:53:55.513261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.513184 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:53:55.618597 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.618549 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" podStartSLOduration=1.61853508 podStartE2EDuration="1.61853508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:55.616902203 +0000 UTC m=+1905.366845192" watchObservedRunningTime="2026-11-01 12:53:55.61853508 +0000 UTC m=+1905.368478058" Nov 01 12:53:55.726116 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:53:55.726062 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" podStartSLOduration=1.726048354 podStartE2EDuration="1.726048354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:55.72250534 +0000 UTC m=+1905.472448317" watchObservedRunningTime="2026-11-01 12:53:55.726048354 +0000 UTC m=+1905.475991333" Nov 01 12:54:00.252293 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:54:00.252252 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:54:00.405790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:00.405753 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:05.978683 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:05.978649 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:06.522953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:06.522921 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:54:06.523103 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:06.522985 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:54:10.545847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:10.545816 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc"] Nov 01 12:54:10.546292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:10.546017 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" podUID="986a09ca-99b6-47cd-a9ba-c5c71fcb070f" containerName="maas-api" containerID="cri-o://c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358" gracePeriod=30 Nov 01 12:54:10.643563 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:10.643505 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:10.888593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:10.883830 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:10.893429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:10.893386 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:54:11.009325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.006574 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\") pod \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " Nov 01 12:54:11.009325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.006745 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs\") pod \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " Nov 01 12:54:11.009325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.006823 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5\" (UniqueName: \"kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5\") pod \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " Nov 01 12:54:11.009325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.006886 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca\") pod \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\" (UID: \"986a09ca-99b6-47cd-a9ba-c5c71fcb070f\") " Nov 01 12:54:11.030331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.018505 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca" pod "986a09ca-99b6-47cd-a9ba-c5c71fcb070f" (UID: "986a09ca-99b6-47cd-a9ba-c5c71fcb070f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:54:11.106388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.106025 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs" pod "986a09ca-99b6-47cd-a9ba-c5c71fcb070f" (UID: "986a09ca-99b6-47cd-a9ba-c5c71fcb070f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:54:11.109434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.109402 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:11.109434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.109434 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:11.112470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.112434 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls" pod "986a09ca-99b6-47cd-a9ba-c5c71fcb070f" (UID: "986a09ca-99b6-47cd-a9ba-c5c71fcb070f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:54:11.113762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.113735 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5" pod "986a09ca-99b6-47cd-a9ba-c5c71fcb070f" (UID: "986a09ca-99b6-47cd-a9ba-c5c71fcb070f"). InnerVolumeSpecName "kube-api-access-6xhv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:11.217789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.217744 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhv5\" (UniqueName: \"kubernetes.io/projected/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-kube-api-access-6xhv5\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:11.217789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.217786 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/986a09ca-99b6-47cd-a9ba-c5c71fcb070f-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:11.622616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.622577 4484 generic.go:356] "Generic (PLEG): container finished" podID="986a09ca-99b6-47cd-a9ba-c5c71fcb070f" containerID="c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358" exitCode=0 Nov 01 12:54:11.623072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.622612 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" event={"ID":"986a09ca-99b6-47cd-a9ba-c5c71fcb070f","Type":"ContainerDied","Data":"c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358"} Nov 01 12:54:11.623072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.622639 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" Nov 01 12:54:11.623072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.622675 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc" event={"ID":"986a09ca-99b6-47cd-a9ba-c5c71fcb070f","Type":"ContainerDied","Data":"b66d31f7a57ee53ef6a305f40ca3dca8a4e239f0d6b79ec44a6def4f4a372290"} Nov 01 12:54:11.623072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.622703 4484 scope.go:122] "RemoveContainer" containerID="c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358" Nov 01 12:54:11.635315 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.635285 4484 scope.go:122] "RemoveContainer" containerID="c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358" Nov 01 12:54:11.635548 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:54:11.635528 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358\": container with ID starting with c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358 not found: ID does not exist" containerID="c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358" Nov 01 12:54:11.635593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.635556 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358"} err="failed to get container status \"c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358\": rpc error: code = NotFound desc = could not find container \"c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358\": container with ID starting with c411a710d99fb73ca3b8dd36750391ea89f2462ed0613642391b50c99565e358 not found: ID does not exist" Nov 01 12:54:11.844324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.841957 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc"] Nov 01 12:54:11.881379 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:11.878176 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47feb945-c9697b5bd-zqgjc"] Nov 01 12:54:12.370215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:12.370175 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986a09ca-99b6-47cd-a9ba-c5c71fcb070f" path="/var/lib/kubelet/pods/986a09ca-99b6-47cd-a9ba-c5c71fcb070f/volumes" Nov 01 12:54:12.432647 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:12.432617 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:12.684793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:12.684719 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:12.809323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:12.808971 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:13.350927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:13.350898 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:15.883588 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:15.883546 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:54:18.474069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:18.474031 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:18.653852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:18.652566 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:18.778903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:18.778842 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:19.816432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:19.816389 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:20.036538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:20.036506 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:20.186394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:20.186291 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:21.254255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:21.254215 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:23.021448 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.021417 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd"] Nov 01 12:54:23.022109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.022091 4484 cpu_manager.go:414] "containerMap: removing container" podUID="986a09ca-99b6-47cd-a9ba-c5c71fcb070f" containerName="maas-api" Nov 01 12:54:23.022109 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.022109 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986a09ca-99b6-47cd-a9ba-c5c71fcb070f" containerName="maas-api" Nov 01 12:54:23.022266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.022253 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="986a09ca-99b6-47cd-a9ba-c5c71fcb070f" containerName="maas-api" Nov 01 12:54:23.027913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.027891 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.077001 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.076975 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0a0610-openshift-default-dockercfg-pzb5v\"" Nov 01 12:54:23.107993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.107953 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108109 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108176 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108288 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjkq\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108466 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108444 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108507 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108569 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108557 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108627 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.108709 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.108685 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.126340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.126296 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd"] Nov 01 12:54:23.130128 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.130097 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220479 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220578 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220623 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220665 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220711 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220757 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220864 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220948 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.222160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.220992 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjkq\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.235912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.235710 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.236486 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.236463 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.237079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.237055 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.271398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.241634 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.271398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.242330 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.271398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.264378 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.271398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.271277 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.312874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.312784 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjkq\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.342342 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.340026 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:23.349029 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.348623 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token\") pod \"e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:23.642175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:23.642067 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:24.071542 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.071506 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd"] Nov 01 12:54:24.076999 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:54:24.076970 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd70cefc_cb18_4e20_a779_3f495184858d.slice/crio-0f0c0759c7691d37802f40454c06d5fc52bb0e829504d8ac07d2827868765733 WatchSource:0}: Error finding container 0f0c0759c7691d37802f40454c06d5fc52bb0e829504d8ac07d2827868765733: Status 404 returned error can't find the container with id 0f0c0759c7691d37802f40454c06d5fc52bb0e829504d8ac07d2827868765733 Nov 01 12:54:24.079775 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.079744 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:24.079841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.079812 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:24.079880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.079842 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:24.394478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.394383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:24.769986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.769949 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" event={"ID":"fd70cefc-cb18-4e20-a779-3f495184858d","Type":"ContainerStarted","Data":"6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21"} Nov 01 12:54:24.770176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.770001 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" event={"ID":"fd70cefc-cb18-4e20-a779-3f495184858d","Type":"ContainerStarted","Data":"0f0c0759c7691d37802f40454c06d5fc52bb0e829504d8ac07d2827868765733"} Nov 01 12:54:24.925604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:24.925552 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podStartSLOduration=2.925539098 podStartE2EDuration="2.925539098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:24.923808285 +0000 UTC m=+1934.673751262" watchObservedRunningTime="2026-11-01 12:54:24.925539098 +0000 UTC m=+1934.675482075" Nov 01 12:54:25.642887 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:25.642853 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:25.648707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:25.648687 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:25.777097 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:25.777068 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:25.777950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:25.777928 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:54:27.676008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:27.675977 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:27.925449 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:27.925411 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:28.057487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:28.057452 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:30.093856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:30.093822 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:30.269733 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:54:30.269693 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:54:31.019422 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.019384 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:31.200330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.193903 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:31.451376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.447494 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f"] Nov 01 12:54:31.456436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.453870 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.501418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.501381 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f"] Nov 01 12:54:31.570347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.546572 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0a0610\"" Nov 01 12:54:31.570347 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.546766 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0a0610-dockercfg-jjrsm\"" Nov 01 12:54:31.652536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.652498 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.735838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.735807 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjmk\" (UniqueName: \"kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk\") pod \"payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.736869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.723342 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj"] Nov 01 12:54:31.783202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.783175 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj"] Nov 01 12:54:31.785339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.783436 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:31.837079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.837052 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:31.837248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.837212 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.837315 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.837244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5w2\" (UniqueName: \"kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2\") pod \"payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:31.837315 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.837276 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjmk\" (UniqueName: \"kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk\") pod \"payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.838082 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.838057 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.893126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.893092 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjmk\" (UniqueName: \"kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk\") pod \"payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:31.938366 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.938339 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5w2\" (UniqueName: \"kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2\") pod \"payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:31.938505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.938465 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:31.939102 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.939082 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:31.991006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:31.990949 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5w2\" (UniqueName: \"kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2\") pod \"payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:32.078364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.078333 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:32.107270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.107242 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:32.461053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.460973 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f"] Nov 01 12:54:32.466546 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:54:32.466516 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdce95a_17d2_40f4_a33d_e4f1af50acef.slice/crio-73269140571b308d28e27b3cf3831b450014c2eaf355f22f63589296c4a2743f WatchSource:0}: Error finding container 73269140571b308d28e27b3cf3831b450014c2eaf355f22f63589296c4a2743f: Status 404 returned error can't find the container with id 73269140571b308d28e27b3cf3831b450014c2eaf355f22f63589296c4a2743f Nov 01 12:54:32.513062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.513037 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj"] Nov 01 12:54:32.518510 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:54:32.518488 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fa3971_744a_4ad4_bd5d_631db925c944.slice/crio-72df339bdbaa465f12f61f13d31967484fc7dff257a312df64b048646df6172c WatchSource:0}: Error finding container 72df339bdbaa465f12f61f13d31967484fc7dff257a312df64b048646df6172c: Status 404 returned error can't find the container with id 72df339bdbaa465f12f61f13d31967484fc7dff257a312df64b048646df6172c Nov 01 12:54:32.830242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.830203 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" event={"ID":"b9fa3971-744a-4ad4-bd5d-631db925c944","Type":"ContainerStarted","Data":"dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb"} Nov 01 12:54:32.830462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.830326 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" event={"ID":"b9fa3971-744a-4ad4-bd5d-631db925c944","Type":"ContainerStarted","Data":"72df339bdbaa465f12f61f13d31967484fc7dff257a312df64b048646df6172c"} Nov 01 12:54:32.830462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.830365 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:32.832129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.832103 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" event={"ID":"acdce95a-17d2-40f4-a33d-e4f1af50acef","Type":"ContainerStarted","Data":"c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673"} Nov 01 12:54:32.832257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.832151 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" event={"ID":"acdce95a-17d2-40f4-a33d-e4f1af50acef","Type":"ContainerStarted","Data":"73269140571b308d28e27b3cf3831b450014c2eaf355f22f63589296c4a2743f"} Nov 01 12:54:32.832257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.832182 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:32.937837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:32.937786 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" podStartSLOduration=1.937772995 podStartE2EDuration="1.937772995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:32.935074484 +0000 UTC m=+1942.685017472" watchObservedRunningTime="2026-11-01 12:54:32.937772995 +0000 UTC m=+1942.687715975" Nov 01 12:54:33.043330 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:33.043265 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" podStartSLOduration=2.043253351 podStartE2EDuration="2.043253351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:33.042922429 +0000 UTC m=+1942.792865408" watchObservedRunningTime="2026-11-01 12:54:33.043253351 +0000 UTC m=+1942.793196329" Nov 01 12:54:35.870425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:35.870388 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm"] Nov 01 12:54:35.870870 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:35.870620 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" containerID="cri-o://eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8" gracePeriod=30 Nov 01 12:54:35.896744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:35.896716 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v"] Nov 01 12:54:35.896914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:35.896896 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" containerID="cri-o://e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5" gracePeriod=30 Nov 01 12:54:36.085099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:36.085064 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t"] Nov 01 12:54:36.085324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:36.085288 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" containerID="cri-o://9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1" gracePeriod=30 Nov 01 12:54:36.092556 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:36.092529 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:38.410922 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.410883 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:54:38.411340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.410962 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:54:38.467492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.467462 4484 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:54:38.467643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.467509 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:54:38.696978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.696895 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:54:38.696978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.696961 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:54:38.887814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:38.887782 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:39.322070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:39.322032 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:39.976410 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:39.973941 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:40.355355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:40.355241 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:41.275671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.275643 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:54:41.280354 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.280332 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:54:41.380385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.380361 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:54:41.384488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384466 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.384573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384527 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.384573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384564 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.384698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384646 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.384762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384713 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.384762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384726 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.384862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384765 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.384862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384811 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.384862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384826 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.385023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384859 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.385023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384916 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.385023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.384996 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.385184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385040 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.385184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385046 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.385184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385128 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.385184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385147 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.385184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385167 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.385447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385197 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert\") pod \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\" (UID: \"83fed01f-aa7e-4698-9b9a-29a8e8f94f89\") " Nov 01 12:54:41.385447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385226 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.385447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385237 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.385447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385372 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.385447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385379 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.385447 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385403 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.385712 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385456 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo\") pod \"d78a0a40-774d-442a-ae7b-11e317071bc3\" (UID: \"d78a0a40-774d-442a-ae7b-11e317071bc3\") " Nov 01 12:54:41.385712 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385578 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.385955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385886 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:54:41.385955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385931 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.386140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.385993 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:54:41.386266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386239 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.386460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386441 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.386573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386557 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.386673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386661 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.386760 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386748 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78a0a40-774d-442a-ae7b-11e317071bc3-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.386851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386840 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.386936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.386923 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.387024 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.387011 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.387123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.387110 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.387211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.387199 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.388174 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.388137 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:41.388281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.388193 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:41.388281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.388246 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "kube-api-access-s929h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:41.388444 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.388406 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "kube-api-access-lx559". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:41.388444 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.388411 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.388838 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.388813 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo" pod "83fed01f-aa7e-4698-9b9a-29a8e8f94f89" (UID: "83fed01f-aa7e-4698-9b9a-29a8e8f94f89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:54:41.389195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.389174 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:54:41.389682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.389663 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy" pod "d78a0a40-774d-442a-ae7b-11e317071bc3" (UID: "d78a0a40-774d-442a-ae7b-11e317071bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.487990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.487960 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488182 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488047 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488182 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488113 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488182 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488145 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488182 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488170 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488230 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488313 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488344 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488401 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token\") pod \"311404f1-fbc2-4e49-94c0-4128cea90c0c\" (UID: \"311404f1-fbc2-4e49-94c0-4128cea90c0c\") " Nov 01 12:54:41.488427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488400 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.488679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488444 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.488679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488566 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.488679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488651 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.488833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.488685 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:54:41.489149 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489128 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489154 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx559\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-kube-api-access-lx559\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489170 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489185 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489199 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s929h\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-kube-api-access-s929h\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489213 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489224 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489233 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489241 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311404f1-fbc2-4e49-94c0-4128cea90c0c-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489250 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489262 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fed01f-aa7e-4698-9b9a-29a8e8f94f89-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489273 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.489274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.489283 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78a0a40-774d-442a-ae7b-11e317071bc3-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.490457 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.490435 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "kube-api-access-2kv7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:41.490536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.490462 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:54:41.490631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.490615 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:54:41.490793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.490779 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token" pod "311404f1-fbc2-4e49-94c0-4128cea90c0c" (UID: "311404f1-fbc2-4e49-94c0-4128cea90c0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:54:41.590559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.590526 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.590559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.590557 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.590773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.590572 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kv7c\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-kube-api-access-2kv7c\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.590773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.590585 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311404f1-fbc2-4e49-94c0-4128cea90c0c-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:54:41.898719 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.898642 4484 generic.go:356] "Generic (PLEG): container finished" podID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerID="e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5" exitCode=0 Nov 01 12:54:41.898719 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.898688 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" Nov 01 12:54:41.898961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.898748 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" event={"ID":"83fed01f-aa7e-4698-9b9a-29a8e8f94f89","Type":"ContainerDied","Data":"e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5"} Nov 01 12:54:41.898961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.898799 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v" event={"ID":"83fed01f-aa7e-4698-9b9a-29a8e8f94f89","Type":"ContainerDied","Data":"286b33c90f8136ac3034a375bb5c5c7c16b413fb2a90aa7803521ec312d4ed6b"} Nov 01 12:54:41.898961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.898831 4484 scope.go:122] "RemoveContainer" containerID="e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5" Nov 01 12:54:41.900364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.900338 4484 generic.go:356] "Generic (PLEG): container finished" podID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerID="eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8" exitCode=0 Nov 01 12:54:41.900507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.900416 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" event={"ID":"d78a0a40-774d-442a-ae7b-11e317071bc3","Type":"ContainerDied","Data":"eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8"} Nov 01 12:54:41.900507 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.900456 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" event={"ID":"d78a0a40-774d-442a-ae7b-11e317071bc3","Type":"ContainerDied","Data":"27b33243a5ea23684315e831dc9682efe90e27a253ee77b292f20239b98d7248"} Nov 01 12:54:41.900627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.900520 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm" Nov 01 12:54:41.904711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.904624 4484 generic.go:356] "Generic (PLEG): container finished" podID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerID="9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1" exitCode=0 Nov 01 12:54:41.904848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.904756 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" event={"ID":"311404f1-fbc2-4e49-94c0-4128cea90c0c","Type":"ContainerDied","Data":"9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1"} Nov 01 12:54:41.904848 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.904794 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" event={"ID":"311404f1-fbc2-4e49-94c0-4128cea90c0c","Type":"ContainerDied","Data":"0722f65f460cee20e8b77db01aa9ed49554f518aa3df5a6599d4f8f9a7407787"} Nov 01 12:54:41.904948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.904871 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t" Nov 01 12:54:41.936565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.936494 4484 scope.go:122] "RemoveContainer" containerID="e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5" Nov 01 12:54:41.936795 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:54:41.936768 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5\": container with ID starting with e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5 not found: ID does not exist" containerID="e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5" Nov 01 12:54:41.936899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.936804 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5"} err="failed to get container status \"e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5\": rpc error: code = NotFound desc = could not find container \"e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5\": container with ID starting with e0fd197d812839c0251524f0c63326b7c4ab8532b655f13d719cf1ad6c34c4d5 not found: ID does not exist" Nov 01 12:54:41.936899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.936826 4484 scope.go:122] "RemoveContainer" containerID="eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8" Nov 01 12:54:41.946442 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.946425 4484 scope.go:122] "RemoveContainer" containerID="eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8" Nov 01 12:54:41.946714 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:54:41.946691 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8\": container with ID starting with eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8 not found: ID does not exist" containerID="eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8" Nov 01 12:54:41.946782 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.946723 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8"} err="failed to get container status \"eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8\": rpc error: code = NotFound desc = could not find container \"eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8\": container with ID starting with eab4fc95e868fa0c2bf7fe873a3be7c7da4d35903ea7fa3af4d80e09ff2629d8 not found: ID does not exist" Nov 01 12:54:41.946782 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.946746 4484 scope.go:122] "RemoveContainer" containerID="9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1" Nov 01 12:54:41.955650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.955622 4484 scope.go:122] "RemoveContainer" containerID="9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1" Nov 01 12:54:41.955891 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:54:41.955875 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1\": container with ID starting with 9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1 not found: ID does not exist" containerID="9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1" Nov 01 12:54:41.955933 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:41.955897 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1"} err="failed to get container status \"9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1\": rpc error: code = NotFound desc = could not find container \"9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1\": container with ID starting with 9715fca176705174f132db61b04beb705856c06cf6b7a880ba122dc7b9abc4b1 not found: ID does not exist" Nov 01 12:54:42.057880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.057850 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v"] Nov 01 12:54:42.061385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.061357 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-f8w7v"] Nov 01 12:54:42.079284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.079261 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:42.187169 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.187030 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm"] Nov 01 12:54:42.192965 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.192931 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-pgnxm"] Nov 01 12:54:42.285561 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.285530 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t"] Nov 01 12:54:42.290376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.290350 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-5jm8t"] Nov 01 12:54:42.371372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.371344 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" path="/var/lib/kubelet/pods/311404f1-fbc2-4e49-94c0-4128cea90c0c/volumes" Nov 01 12:54:42.371874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.371859 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" path="/var/lib/kubelet/pods/83fed01f-aa7e-4698-9b9a-29a8e8f94f89/volumes" Nov 01 12:54:42.372289 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:42.372276 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" path="/var/lib/kubelet/pods/d78a0a40-774d-442a-ae7b-11e317071bc3/volumes" Nov 01 12:54:43.693763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:43.693711 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:43.847050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:43.847023 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:54:43.847232 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:43.847099 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:54:43.881998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:43.881961 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:44.009426 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:44.009397 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:47.917215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:47.917182 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:51.167701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.167669 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:51.435728 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.435637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:51.617324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.613001 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:51.930728 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.930685 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:51.957472 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.957442 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm"] Nov 01 12:54:51.960156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.960129 4484 cpu_manager.go:414] "containerMap: removing container" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" Nov 01 12:54:51.961170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.961152 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" Nov 01 12:54:51.961423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.961409 4484 cpu_manager.go:414] "containerMap: removing container" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" Nov 01 12:54:51.961746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.961730 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" Nov 01 12:54:51.961903 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.961891 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" Nov 01 12:54:51.981138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:51.961999 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" Nov 01 12:54:52.238783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.238530 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="311404f1-fbc2-4e49-94c0-4128cea90c0c" containerName="istio-proxy" Nov 01 12:54:52.238783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.238606 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fed01f-aa7e-4698-9b9a-29a8e8f94f89" containerName="istio-proxy" Nov 01 12:54:52.238783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.238624 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78a0a40-774d-442a-ae7b-11e317071bc3" containerName="istio-proxy" Nov 01 12:54:52.286593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.286564 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm"] Nov 01 12:54:52.286963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.286946 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.297358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.297338 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph"] Nov 01 12:54:52.302845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.302822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph"] Nov 01 12:54:52.302949 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.302914 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.336476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.336441 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-78c69d\"" Nov 01 12:54:52.336476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.336469 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-78c69d-dockercfg-s2jmf\"" Nov 01 12:54:52.384026 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.383991 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvqz\" (UniqueName: \"kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz\") pod \"payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.384176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.384040 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.384334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.384309 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2qt\" (UniqueName: \"kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt\") pod \"payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.384414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.384350 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.485730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.485669 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2qt\" (UniqueName: \"kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt\") pod \"payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.485959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.485737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.485959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.485862 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvqz\" (UniqueName: \"kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz\") pod \"payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.485959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.485948 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.486626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.486596 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.486842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.486814 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.538639 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.538575 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvqz\" (UniqueName: \"kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz\") pod \"payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:52.538840 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.538818 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2qt\" (UniqueName: \"kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt\") pod \"payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.572346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.572294 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:52.607913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.607887 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:52.619526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:52.619500 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:53.003798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.003766 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm"] Nov 01 12:54:53.038448 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:54:53.038418 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4552e6_6af3_4009_8691_82f08f571deb.slice/crio-a9900a764118b92961cfcaa0f2528f383cde610a2d09401ef16fe879fc846291 WatchSource:0}: Error finding container a9900a764118b92961cfcaa0f2528f383cde610a2d09401ef16fe879fc846291: Status 404 returned error can't find the container with id a9900a764118b92961cfcaa0f2528f383cde610a2d09401ef16fe879fc846291 Nov 01 12:54:53.044204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.044178 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph"] Nov 01 12:54:53.051608 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:54:53.051582 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e1c795_52e6_4891_bc9c_4d696f87dfdf.slice/crio-62ae076c286192204bcdbc9fa1cfccf51071bf9ee4a5dd9540b626d498524e79 WatchSource:0}: Error finding container 62ae076c286192204bcdbc9fa1cfccf51071bf9ee4a5dd9540b626d498524e79: Status 404 returned error can't find the container with id 62ae076c286192204bcdbc9fa1cfccf51071bf9ee4a5dd9540b626d498524e79 Nov 01 12:54:53.300445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.300411 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" event={"ID":"cc4552e6-6af3-4009-8691-82f08f571deb","Type":"ContainerStarted","Data":"2d3f684e4397769e64121f5f5a58122bba5bb8c35b4af3b62b7a393eef2b8f0b"} Nov 01 12:54:53.300830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.300475 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" event={"ID":"cc4552e6-6af3-4009-8691-82f08f571deb","Type":"ContainerStarted","Data":"a9900a764118b92961cfcaa0f2528f383cde610a2d09401ef16fe879fc846291"} Nov 01 12:54:53.300830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.300522 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:54:53.302988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.302967 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" event={"ID":"60e1c795-52e6-4891-bc9c-4d696f87dfdf","Type":"ContainerStarted","Data":"62ae076c286192204bcdbc9fa1cfccf51071bf9ee4a5dd9540b626d498524e79"} Nov 01 12:54:53.409835 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.409787 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" podStartSLOduration=2.409772578 podStartE2EDuration="2.409772578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:53.406907573 +0000 UTC m=+1963.156850550" watchObservedRunningTime="2026-11-01 12:54:53.409772578 +0000 UTC m=+1963.159715556" Nov 01 12:54:53.784679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:53.784645 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:54.229453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.229415 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq"] Nov 01 12:54:54.235254 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.235230 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.284473 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.284441 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2810c2-openshift-default-dockercfg-wddvh\"" Nov 01 12:54:54.316342 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.314500 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:54.318440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.318411 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" event={"ID":"60e1c795-52e6-4891-bc9c-4d696f87dfdf","Type":"ContainerStarted","Data":"c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539"} Nov 01 12:54:54.318584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.318462 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:54:54.347991 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.347950 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2dq\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348005 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348176 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348334 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348389 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348467 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348527 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348562 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.348605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.348593 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.459997 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.459935 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.460270 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.460245 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.460529 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.460511 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.460661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.460648 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.460754 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.460742 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.460854 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.460840 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.461118 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.461104 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2dq\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.461223 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.461210 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.461387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.461373 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.463181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.463151 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.463535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.463506 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.463707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.463687 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.473740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.464246 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.473740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.464495 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.473740 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.466647 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.486346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.486250 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.494102 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.494075 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq"] Nov 01 12:54:54.496164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.496108 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" podStartSLOduration=3.496090794 podStartE2EDuration="3.496090794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:54.481527388 +0000 UTC m=+1964.231470367" watchObservedRunningTime="2026-11-01 12:54:54.496090794 +0000 UTC m=+1964.246033773" Nov 01 12:54:54.518369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.518344 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2dq\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.519538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.519509 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token\") pod \"e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.550664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.550637 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:54.664359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.664226 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:54.988907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.988875 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq"] Nov 01 12:54:54.995854 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:54:54.995822 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc4d29d_9be9_42e5_a95a_59df4aa746f0.slice/crio-cf04e0a494cbcb87fb07cfe6db391f0e6696e5031caa8ae9c09f08b7b21b7dbd WatchSource:0}: Error finding container cf04e0a494cbcb87fb07cfe6db391f0e6696e5031caa8ae9c09f08b7b21b7dbd: Status 404 returned error can't find the container with id cf04e0a494cbcb87fb07cfe6db391f0e6696e5031caa8ae9c09f08b7b21b7dbd Nov 01 12:54:54.999071 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.999028 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:54.999170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.999109 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:54.999170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:54.999158 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:55.326950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:55.326859 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" event={"ID":"4fc4d29d-9be9-42e5-a95a-59df4aa746f0","Type":"ContainerStarted","Data":"2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7"} Nov 01 12:54:55.326950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:55.326919 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" event={"ID":"4fc4d29d-9be9-42e5-a95a-59df4aa746f0","Type":"ContainerStarted","Data":"cf04e0a494cbcb87fb07cfe6db391f0e6696e5031caa8ae9c09f08b7b21b7dbd"} Nov 01 12:54:55.440213 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:55.440158 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podStartSLOduration=1.440143617 podStartE2EDuration="1.440143617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:55.437295914 +0000 UTC m=+1965.187238892" watchObservedRunningTime="2026-11-01 12:54:55.440143617 +0000 UTC m=+1965.190086599" Nov 01 12:54:55.551357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:55.551327 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:56.557032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:56.557000 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:57.343882 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.343844 4484 generic.go:356] "Generic (PLEG): container finished" podID="baae4295-9113-41d7-a6db-1d030f4590e5" containerID="78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82" exitCode=0 Nov 01 12:54:57.344057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.343918 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" event={"ID":"baae4295-9113-41d7-a6db-1d030f4590e5","Type":"ContainerDied","Data":"78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82"} Nov 01 12:54:57.344190 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.344165 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:57.344513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.344494 4484 scope.go:122] "RemoveContainer" containerID="78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82" Nov 01 12:54:57.345483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.345355 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:54:57.346226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.346195 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:57.346339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.346272 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:57.346339 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:57.346330 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:54:58.237619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:58.237590 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:58.393168 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:58.393133 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" event={"ID":"baae4295-9113-41d7-a6db-1d030f4590e5","Type":"ContainerStarted","Data":"d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685"} Nov 01 12:54:58.424796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:58.424768 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:58.567938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:58.567836 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:59.307510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:59.307478 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:54:59.308381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:59.308352 4484 patch_prober.go:29] interesting pod/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 12:54:59.308492 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:59.308414 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 12:54:59.533659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:59.533622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:54:59.985800 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:59.985770 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g"] Nov 01 12:54:59.990250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:54:59.990230 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.055337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.043204 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g"] Nov 01 12:55:00.078264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.078225 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.078501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.078436 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.078501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.078488 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhh9\" (UniqueName: \"kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.084495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.084467 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:55:00.084627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.084551 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:55:00.179708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.179673 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.179922 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.179728 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhh9\" (UniqueName: \"kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.179922 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.179851 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.180514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.180493 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.182140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.182114 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.232463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.232439 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhh9\" (UniqueName: \"kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.274892 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:55:00.274790 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:55:00.303652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.303626 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:00.307917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.307894 4484 patch_prober.go:29] interesting pod/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 12:55:00.308148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.307944 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 12:55:00.707319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:00.707233 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g"] Nov 01 12:55:00.712626 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:55:00.712597 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e19db5d_9109_4050_a2cc_b91fd5aa848b.slice/crio-3bd07562ee6b78a54c3bef51431b3877ba71b8ee0d0d8d2364f9ee815d2f3783 WatchSource:0}: Error finding container 3bd07562ee6b78a54c3bef51431b3877ba71b8ee0d0d8d2364f9ee815d2f3783: Status 404 returned error can't find the container with id 3bd07562ee6b78a54c3bef51431b3877ba71b8ee0d0d8d2364f9ee815d2f3783 Nov 01 12:55:01.307593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:01.307559 4484 patch_prober.go:29] interesting pod/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 12:55:01.307762 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:01.307610 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 12:55:01.394919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:01.394884 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" event={"ID":"9e19db5d-9109-4050-a2cc-b91fd5aa848b","Type":"ContainerStarted","Data":"581b7380508df298f220e12780104170369681e5671e7f6a6d41e84ad876eb60"} Nov 01 12:55:01.395412 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:01.394936 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" event={"ID":"9e19db5d-9109-4050-a2cc-b91fd5aa848b","Type":"ContainerStarted","Data":"3bd07562ee6b78a54c3bef51431b3877ba71b8ee0d0d8d2364f9ee815d2f3783"} Nov 01 12:55:01.527707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:01.527650 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" podStartSLOduration=2.527634233 podStartE2EDuration="2.527634233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:01.52425901 +0000 UTC m=+1971.274201988" watchObservedRunningTime="2026-11-01 12:55:01.527634233 +0000 UTC m=+1971.277577211" Nov 01 12:55:02.307846 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:02.307813 4484 patch_prober.go:29] interesting pod/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 12:55:02.308014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:02.307869 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 12:55:02.404619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:02.404583 4484 generic.go:356] "Generic (PLEG): container finished" podID="9e19db5d-9109-4050-a2cc-b91fd5aa848b" containerID="581b7380508df298f220e12780104170369681e5671e7f6a6d41e84ad876eb60" exitCode=0 Nov 01 12:55:02.405012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:02.404648 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" event={"ID":"9e19db5d-9109-4050-a2cc-b91fd5aa848b","Type":"ContainerDied","Data":"581b7380508df298f220e12780104170369681e5671e7f6a6d41e84ad876eb60"} Nov 01 12:55:03.307284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.307250 4484 patch_prober.go:29] interesting pod/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 12:55:03.307485 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.307310 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 12:55:03.617767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.617742 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:03.729164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.729131 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca\") pod \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " Nov 01 12:55:03.729164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.729181 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca\") pod \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " Nov 01 12:55:03.729648 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.729369 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9\" (UniqueName: \"kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9\") pod \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\" (UID: \"9e19db5d-9109-4050-a2cc-b91fd5aa848b\") " Nov 01 12:55:03.730706 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.730671 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca" pod "9e19db5d-9109-4050-a2cc-b91fd5aa848b" (UID: "9e19db5d-9109-4050-a2cc-b91fd5aa848b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:55:03.732959 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.732933 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca" pod "9e19db5d-9109-4050-a2cc-b91fd5aa848b" (UID: "9e19db5d-9109-4050-a2cc-b91fd5aa848b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:55:03.734058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.734029 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9" pod "9e19db5d-9109-4050-a2cc-b91fd5aa848b" (UID: "9e19db5d-9109-4050-a2cc-b91fd5aa848b"). InnerVolumeSpecName "kube-api-access-7fhh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:55:03.831779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.831747 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:03.831779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.831776 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e19db5d-9109-4050-a2cc-b91fd5aa848b-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:03.831779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:03.831786 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fhh9\" (UniqueName: \"kubernetes.io/projected/9e19db5d-9109-4050-a2cc-b91fd5aa848b-kube-api-access-7fhh9\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:04.312340 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.312293 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:55:04.312705 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.312683 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:55:04.313346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.313325 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:55:04.321226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.321207 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:55:04.419877 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.419850 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" Nov 01 12:55:04.420034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.419891 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g" event={"ID":"9e19db5d-9109-4050-a2cc-b91fd5aa848b","Type":"ContainerDied","Data":"3bd07562ee6b78a54c3bef51431b3877ba71b8ee0d0d8d2364f9ee815d2f3783"} Nov 01 12:55:04.420034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:04.419942 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bd07562ee6b78a54c3bef51431b3877ba71b8ee0d0d8d2364f9ee815d2f3783" Nov 01 12:55:05.424951 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:05.424917 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:06.337644 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:06.337612 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:55:08.115377 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.115339 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:08.346617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.344537 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:08.685864 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.685832 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv"] Nov 01 12:55:08.686595 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.686575 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9e19db5d-9109-4050-a2cc-b91fd5aa848b" containerName="revoke-keys" Nov 01 12:55:08.686715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.686595 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e19db5d-9109-4050-a2cc-b91fd5aa848b" containerName="revoke-keys" Nov 01 12:55:08.686796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.686782 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e19db5d-9109-4050-a2cc-b91fd5aa848b" containerName="revoke-keys" Nov 01 12:55:08.690375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.690355 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:08.744324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.738832 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2810c2\"" Nov 01 12:55:08.926621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.926554 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrfrw\" (UniqueName: \"kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw\") pod \"payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:08.926855 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.926832 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:08.933180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.931350 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2810c2-dockercfg-vnt58\"" Nov 01 12:55:08.940752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.940349 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv"] Nov 01 12:55:08.951196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.951166 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x"] Nov 01 12:55:08.963314 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.962902 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:08.997773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:08.997749 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x"] Nov 01 12:55:09.027776 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.027745 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrfrw\" (UniqueName: \"kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw\") pod \"payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:09.027919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.027903 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:09.028707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.028689 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:09.081153 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.081129 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrfrw\" (UniqueName: \"kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw\") pod \"payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:09.129356 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.129293 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.129683 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.129518 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npkss\" (UniqueName: \"kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss\") pod \"payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.230901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.230877 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.231041 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.231009 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npkss\" (UniqueName: \"kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss\") pod \"payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.231595 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.231573 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.284649 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.284622 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npkss\" (UniqueName: \"kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss\") pod \"payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.297094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.297054 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:09.304352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.304331 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:09.681160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.681082 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x"] Nov 01 12:55:09.685039 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:55:09.685013 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b169ce6_fd56_4dd5_ac66_747913843465.slice/crio-c96742a2852fb53ae83d8966efe6bf6b70d785b8ce0eb0fe0239b6802b098ba2 WatchSource:0}: Error finding container c96742a2852fb53ae83d8966efe6bf6b70d785b8ce0eb0fe0239b6802b098ba2: Status 404 returned error can't find the container with id c96742a2852fb53ae83d8966efe6bf6b70d785b8ce0eb0fe0239b6802b098ba2 Nov 01 12:55:09.731123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:09.731095 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv"] Nov 01 12:55:09.749281 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:55:09.749251 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcd8b6e_1628_4d90_aba1_fb9adc04bcac.slice/crio-f62a475dd383fef394b9068e7033d35000f5381ec71e9abaf4889de48880e45f WatchSource:0}: Error finding container f62a475dd383fef394b9068e7033d35000f5381ec71e9abaf4889de48880e45f: Status 404 returned error can't find the container with id f62a475dd383fef394b9068e7033d35000f5381ec71e9abaf4889de48880e45f Nov 01 12:55:10.527378 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.527343 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" event={"ID":"6b169ce6-fd56-4dd5-ac66-747913843465","Type":"ContainerStarted","Data":"8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320"} Nov 01 12:55:10.527861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.527397 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" event={"ID":"6b169ce6-fd56-4dd5-ac66-747913843465","Type":"ContainerStarted","Data":"c96742a2852fb53ae83d8966efe6bf6b70d785b8ce0eb0fe0239b6802b098ba2"} Nov 01 12:55:10.527861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.527555 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:10.529199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.529173 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" event={"ID":"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac","Type":"ContainerStarted","Data":"4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20"} Nov 01 12:55:10.529337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.529224 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:10.529337 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.529240 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" event={"ID":"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac","Type":"ContainerStarted","Data":"f62a475dd383fef394b9068e7033d35000f5381ec71e9abaf4889de48880e45f"} Nov 01 12:55:10.634921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.634864 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" podStartSLOduration=2.634845728 podStartE2EDuration="2.634845728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:10.631580236 +0000 UTC m=+1980.381523209" watchObservedRunningTime="2026-11-01 12:55:10.634845728 +0000 UTC m=+1980.384788708" Nov 01 12:55:10.742108 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:10.742051 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" podStartSLOduration=2.742035322 podStartE2EDuration="2.742035322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:10.738007438 +0000 UTC m=+1980.487950417" watchObservedRunningTime="2026-11-01 12:55:10.742035322 +0000 UTC m=+1980.491978301" Nov 01 12:55:11.154913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:11.154876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:11.271682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:11.271652 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:11.384667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:11.384640 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:15.213750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:15.213718 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw"] Nov 01 12:55:15.319271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:15.319237 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl"] Nov 01 12:55:15.562831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:15.562793 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:15.723906 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:15.723867 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:16.998137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:16.998103 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:20.214602 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.214549 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" podUID="d7725753-6369-44c9-bfdd-ed87b011d181" containerName="payload-processing" containerID="cri-o://28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d" gracePeriod=25 Nov 01 12:55:20.320521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.320472 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" podUID="93fcb15f-20a0-49ec-a402-b983b05f66f0" containerName="payload-pre-processing" containerID="cri-o://3f1dc7253b17e2e2d5b91a1a336027d2ed1d7ae85ae9352c83df15689587285e" gracePeriod=25 Nov 01 12:55:20.481888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.481856 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:55:20.586198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.586168 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume\") pod \"d7725753-6369-44c9-bfdd-ed87b011d181\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " Nov 01 12:55:20.586423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.586332 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j\" (UniqueName: \"kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j\") pod \"d7725753-6369-44c9-bfdd-ed87b011d181\" (UID: \"d7725753-6369-44c9-bfdd-ed87b011d181\") " Nov 01 12:55:20.586631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.586582 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume" pod "d7725753-6369-44c9-bfdd-ed87b011d181" (UID: "d7725753-6369-44c9-bfdd-ed87b011d181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:55:20.587026 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.586983 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7725753-6369-44c9-bfdd-ed87b011d181-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:20.588438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.588415 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j" pod "d7725753-6369-44c9-bfdd-ed87b011d181" (UID: "d7725753-6369-44c9-bfdd-ed87b011d181"). InnerVolumeSpecName "kube-api-access-rlt4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:55:20.603536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.603512 4484 generic.go:356] "Generic (PLEG): container finished" podID="93fcb15f-20a0-49ec-a402-b983b05f66f0" containerID="3f1dc7253b17e2e2d5b91a1a336027d2ed1d7ae85ae9352c83df15689587285e" exitCode=0 Nov 01 12:55:20.603660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.603582 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" event={"ID":"93fcb15f-20a0-49ec-a402-b983b05f66f0","Type":"ContainerDied","Data":"3f1dc7253b17e2e2d5b91a1a336027d2ed1d7ae85ae9352c83df15689587285e"} Nov 01 12:55:20.603660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.603637 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" event={"ID":"93fcb15f-20a0-49ec-a402-b983b05f66f0","Type":"ContainerDied","Data":"881378186e8e8ae2883191144c242bc0723dc39c4748bedfdb097f6e1d266950"} Nov 01 12:55:20.603770 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.603663 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881378186e8e8ae2883191144c242bc0723dc39c4748bedfdb097f6e1d266950" Nov 01 12:55:20.610527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.610504 4484 generic.go:356] "Generic (PLEG): container finished" podID="d7725753-6369-44c9-bfdd-ed87b011d181" containerID="28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d" exitCode=0 Nov 01 12:55:20.610633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.610562 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" Nov 01 12:55:20.610633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.610592 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" event={"ID":"d7725753-6369-44c9-bfdd-ed87b011d181","Type":"ContainerDied","Data":"28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d"} Nov 01 12:55:20.610720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.610636 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw" event={"ID":"d7725753-6369-44c9-bfdd-ed87b011d181","Type":"ContainerDied","Data":"4128750bb4e4f536aea95e0f7661c7969a94fbfa28316042db818e921bf221bb"} Nov 01 12:55:20.610720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.610660 4484 scope.go:122] "RemoveContainer" containerID="28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d" Nov 01 12:55:20.612894 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.612872 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:55:20.620783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.620768 4484 scope.go:122] "RemoveContainer" containerID="28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d" Nov 01 12:55:20.621098 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:55:20.621078 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d\": container with ID starting with 28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d not found: ID does not exist" containerID="28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d" Nov 01 12:55:20.621175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.621108 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d"} err="failed to get container status \"28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d\": rpc error: code = NotFound desc = could not find container \"28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d\": container with ID starting with 28b0eb4d865371b12f59667bb17ba1d71de0ee0d79819185e2e20d9de407ae3d not found: ID does not exist" Nov 01 12:55:20.688480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.688456 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume\") pod \"93fcb15f-20a0-49ec-a402-b983b05f66f0\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " Nov 01 12:55:20.688585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.688547 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927\" (UniqueName: \"kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927\") pod \"93fcb15f-20a0-49ec-a402-b983b05f66f0\" (UID: \"93fcb15f-20a0-49ec-a402-b983b05f66f0\") " Nov 01 12:55:20.688786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.688766 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume" pod "93fcb15f-20a0-49ec-a402-b983b05f66f0" (UID: "93fcb15f-20a0-49ec-a402-b983b05f66f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:55:20.689039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.689020 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlt4j\" (UniqueName: \"kubernetes.io/projected/d7725753-6369-44c9-bfdd-ed87b011d181-kube-api-access-rlt4j\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:20.689039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.689039 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fcb15f-20a0-49ec-a402-b983b05f66f0-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:20.690547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.690526 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927" pod "93fcb15f-20a0-49ec-a402-b983b05f66f0" (UID: "93fcb15f-20a0-49ec-a402-b983b05f66f0"). InnerVolumeSpecName "kube-api-access-pm927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:55:20.790541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.790480 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm927\" (UniqueName: \"kubernetes.io/projected/93fcb15f-20a0-49ec-a402-b983b05f66f0-kube-api-access-pm927\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:20.871791 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.871754 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw"] Nov 01 12:55:20.877115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:20.877079 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f6b043-7d65dd56cf-6rxcw"] Nov 01 12:55:21.540425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:21.540340 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 12:55:21.540425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:21.540412 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 12:55:21.618381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:21.618350 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl" Nov 01 12:55:21.854417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:21.854335 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl"] Nov 01 12:55:21.857660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:21.857633 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f6b043-7d7cc47c9-kpgwl"] Nov 01 12:55:22.371316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:22.371270 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fcb15f-20a0-49ec-a402-b983b05f66f0" path="/var/lib/kubelet/pods/93fcb15f-20a0-49ec-a402-b983b05f66f0/volumes" Nov 01 12:55:22.371784 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:22.371767 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7725753-6369-44c9-bfdd-ed87b011d181" path="/var/lib/kubelet/pods/d7725753-6369-44c9-bfdd-ed87b011d181/volumes" Nov 01 12:55:22.846786 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:22.846719 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:22.966312 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:22.966277 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:23.392028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:23.391995 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:25.510148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:25.510116 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:55:28.846554 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:28.846520 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:30.252955 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:55:30.252918 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:55:31.195723 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:31.195696 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:31.307516 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:31.307488 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:34.200438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:34.200406 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:37.508799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:37.508768 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:39.490802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:39.490766 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:42.596154 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.596116 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr"] Nov 01 12:55:42.596883 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.596862 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d7725753-6369-44c9-bfdd-ed87b011d181" containerName="payload-processing" Nov 01 12:55:42.596927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.596886 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7725753-6369-44c9-bfdd-ed87b011d181" containerName="payload-processing" Nov 01 12:55:42.596927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.596919 4484 cpu_manager.go:414] "containerMap: removing container" podUID="93fcb15f-20a0-49ec-a402-b983b05f66f0" containerName="payload-pre-processing" Nov 01 12:55:42.596995 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.596927 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fcb15f-20a0-49ec-a402-b983b05f66f0" containerName="payload-pre-processing" Nov 01 12:55:42.597134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.597121 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7725753-6369-44c9-bfdd-ed87b011d181" containerName="payload-processing" Nov 01 12:55:42.597179 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.597138 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fcb15f-20a0-49ec-a402-b983b05f66f0" containerName="payload-pre-processing" Nov 01 12:55:42.601478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.601456 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.653256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.653230 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr"] Nov 01 12:55:42.693918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.693893 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:55:42.694105 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.694089 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:55:42.731619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.731587 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.731746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.731648 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.731857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.731841 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28j9\" (UniqueName: \"kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.833502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.833473 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.833666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.833538 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.833743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.833703 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28j9\" (UniqueName: \"kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.834167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.834144 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.835826 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.835804 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.887023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.886976 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28j9\" (UniqueName: \"kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-998lr\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:42.918147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:42.918127 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:43.297130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:43.297095 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr"] Nov 01 12:55:43.302898 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:55:43.302868 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d957e0_ef5c_4c66_a910_cde0deb006c4.slice/crio-68dd61236db1d7b85a95e9e1dbde82cbafec554f35c730c5a2c68cc2e27e1491 WatchSource:0}: Error finding container 68dd61236db1d7b85a95e9e1dbde82cbafec554f35c730c5a2c68cc2e27e1491: Status 404 returned error can't find the container with id 68dd61236db1d7b85a95e9e1dbde82cbafec554f35c730c5a2c68cc2e27e1491 Nov 01 12:55:43.784068 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:43.784043 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" event={"ID":"72d957e0-ef5c-4c66-a910-cde0deb006c4","Type":"ContainerStarted","Data":"6351f6309f47003338abcfce12b51cb5af4444650785e5d2404de4fd475a0595"} Nov 01 12:55:43.784388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:43.784096 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" event={"ID":"72d957e0-ef5c-4c66-a910-cde0deb006c4","Type":"ContainerStarted","Data":"68dd61236db1d7b85a95e9e1dbde82cbafec554f35c730c5a2c68cc2e27e1491"} Nov 01 12:55:43.893996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:43.893944 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" podStartSLOduration=1.8939308609999999 podStartE2EDuration="1.893930861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:43.893589392 +0000 UTC m=+2013.643532371" watchObservedRunningTime="2026-11-01 12:55:43.893930861 +0000 UTC m=+2013.643873875" Nov 01 12:55:44.792175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:44.792144 4484 generic.go:356] "Generic (PLEG): container finished" podID="72d957e0-ef5c-4c66-a910-cde0deb006c4" containerID="6351f6309f47003338abcfce12b51cb5af4444650785e5d2404de4fd475a0595" exitCode=0 Nov 01 12:55:44.792595 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:44.792233 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" event={"ID":"72d957e0-ef5c-4c66-a910-cde0deb006c4","Type":"ContainerDied","Data":"6351f6309f47003338abcfce12b51cb5af4444650785e5d2404de4fd475a0595"} Nov 01 12:55:45.349136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:45.349106 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:45.975099 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:45.975076 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:46.076348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.076282 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca\") pod \"72d957e0-ef5c-4c66-a910-cde0deb006c4\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " Nov 01 12:55:46.076465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.076380 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca\") pod \"72d957e0-ef5c-4c66-a910-cde0deb006c4\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " Nov 01 12:55:46.076532 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.076518 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9\" (UniqueName: \"kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9\") pod \"72d957e0-ef5c-4c66-a910-cde0deb006c4\" (UID: \"72d957e0-ef5c-4c66-a910-cde0deb006c4\") " Nov 01 12:55:46.076686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.076662 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca" pod "72d957e0-ef5c-4c66-a910-cde0deb006c4" (UID: "72d957e0-ef5c-4c66-a910-cde0deb006c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:55:46.076978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.076961 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:46.078414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.078393 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca" pod "72d957e0-ef5c-4c66-a910-cde0deb006c4" (UID: "72d957e0-ef5c-4c66-a910-cde0deb006c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:55:46.078510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.078476 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9" pod "72d957e0-ef5c-4c66-a910-cde0deb006c4" (UID: "72d957e0-ef5c-4c66-a910-cde0deb006c4"). InnerVolumeSpecName "kube-api-access-c28j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:55:46.178737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.178704 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d957e0-ef5c-4c66-a910-cde0deb006c4-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:46.178737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.178733 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28j9\" (UniqueName: \"kubernetes.io/projected/72d957e0-ef5c-4c66-a910-cde0deb006c4-kube-api-access-c28j9\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:46.808388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.808358 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" Nov 01 12:55:46.808388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.808371 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr" event={"ID":"72d957e0-ef5c-4c66-a910-cde0deb006c4","Type":"ContainerDied","Data":"68dd61236db1d7b85a95e9e1dbde82cbafec554f35c730c5a2c68cc2e27e1491"} Nov 01 12:55:46.808606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:46.808426 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68dd61236db1d7b85a95e9e1dbde82cbafec554f35c730c5a2c68cc2e27e1491" Nov 01 12:55:51.233834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:51.233798 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:55.221279 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.221202 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:55.416635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.416598 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:55.570869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.570823 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd"] Nov 01 12:55:55.571910 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.571880 4484 cpu_manager.go:414] "containerMap: removing container" podUID="72d957e0-ef5c-4c66-a910-cde0deb006c4" containerName="revoke-keys" Nov 01 12:55:55.571910 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.571909 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d957e0-ef5c-4c66-a910-cde0deb006c4" containerName="revoke-keys" Nov 01 12:55:55.572175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.572159 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d957e0-ef5c-4c66-a910-cde0deb006c4" containerName="revoke-keys" Nov 01 12:55:55.576030 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.576013 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.605255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.605224 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd"] Nov 01 12:55:55.665319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.664591 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:55:55.665319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.664691 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:55:55.668678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.668258 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.668678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.668319 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4dq\" (UniqueName: \"kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.668678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.668579 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.770073 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.770034 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.770073 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.770077 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4dq\" (UniqueName: \"kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.770362 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.770230 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.770895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.770863 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.772517 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.772493 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.823102 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.823045 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4dq\" (UniqueName: \"kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq\") pod \"maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:55.889553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:55.889512 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:56.286023 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:56.285982 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd"] Nov 01 12:55:56.290950 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:55:56.290924 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63ad19f_2eae_4ff1_8b1d_2a6173622cdd.slice/crio-9bc0ce887ffeaf0383466b81e091a83ab931b867bd502eed6101821c5ecb382f WatchSource:0}: Error finding container 9bc0ce887ffeaf0383466b81e091a83ab931b867bd502eed6101821c5ecb382f: Status 404 returned error can't find the container with id 9bc0ce887ffeaf0383466b81e091a83ab931b867bd502eed6101821c5ecb382f Nov 01 12:55:56.904593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:56.904508 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" event={"ID":"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd","Type":"ContainerStarted","Data":"65779fa790e1456112b60bf843496ec3fc4a845691c656258016fd9cd4a63b7a"} Nov 01 12:55:56.904593 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:56.904569 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" event={"ID":"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd","Type":"ContainerStarted","Data":"9bc0ce887ffeaf0383466b81e091a83ab931b867bd502eed6101821c5ecb382f"} Nov 01 12:55:56.990372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:56.990338 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:55:57.013193 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:57.013141 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" podStartSLOduration=2.013124137 podStartE2EDuration="2.013124137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:57.009843837 +0000 UTC m=+2026.759786815" watchObservedRunningTime="2026-11-01 12:55:57.013124137 +0000 UTC m=+2026.763067118" Nov 01 12:55:57.914444 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:57.914406 4484 generic.go:356] "Generic (PLEG): container finished" podID="c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" containerID="65779fa790e1456112b60bf843496ec3fc4a845691c656258016fd9cd4a63b7a" exitCode=0 Nov 01 12:55:57.914860 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:57.914494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" event={"ID":"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd","Type":"ContainerDied","Data":"65779fa790e1456112b60bf843496ec3fc4a845691c656258016fd9cd4a63b7a"} Nov 01 12:55:59.082627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.082594 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:59.256357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.256322 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq\" (UniqueName: \"kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq\") pod \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " Nov 01 12:55:59.256551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.256439 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca\") pod \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " Nov 01 12:55:59.256551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.256520 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca\") pod \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\" (UID: \"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd\") " Nov 01 12:55:59.256978 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.256946 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca" pod "c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" (UID: "c63ad19f-2eae-4ff1-8b1d-2a6173622cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:55:59.257211 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.257184 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:59.258625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.258601 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca" pod "c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" (UID: "c63ad19f-2eae-4ff1-8b1d-2a6173622cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:55:59.258930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.258908 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq" pod "c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" (UID: "c63ad19f-2eae-4ff1-8b1d-2a6173622cdd"). InnerVolumeSpecName "kube-api-access-8c4dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:55:59.358282 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.358245 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4dq\" (UniqueName: \"kubernetes.io/projected/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-kube-api-access-8c4dq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:59.358282 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.358280 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:55:59.931850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.931806 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" Nov 01 12:55:59.931850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.931832 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd" event={"ID":"c63ad19f-2eae-4ff1-8b1d-2a6173622cdd","Type":"ContainerDied","Data":"9bc0ce887ffeaf0383466b81e091a83ab931b867bd502eed6101821c5ecb382f"} Nov 01 12:55:59.932126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:55:59.931885 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc0ce887ffeaf0383466b81e091a83ab931b867bd502eed6101821c5ecb382f" Nov 01 12:56:00.252099 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:56:00.252065 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:56:00.940080 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:00.940047 4484 generic.go:356] "Generic (PLEG): container finished" podID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerID="0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107" exitCode=0 Nov 01 12:56:00.940262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:00.940093 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" event={"ID":"11835b7f-717d-48fa-8971-65b79dfa49d9","Type":"ContainerDied","Data":"0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107"} Nov 01 12:56:00.940479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:00.940457 4484 scope.go:122] "RemoveContainer" containerID="0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107" Nov 01 12:56:00.942131 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:00.942100 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:56:00.942200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:00.942177 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:56:00.942237 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:00.942210 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:56:01.904326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:01.898162 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:02.060323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:02.059479 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" event={"ID":"11835b7f-717d-48fa-8971-65b79dfa49d9","Type":"ContainerStarted","Data":"8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64"} Nov 01 12:56:02.889319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:02.889025 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:56:02.889319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:02.889151 4484 patch_prober.go:29] interesting pod/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 12:56:02.889319 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:02.889196 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 12:56:03.428778 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.428745 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:56:03.530219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.530189 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb"] Nov 01 12:56:03.530856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.530842 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" containerName="revoke-keys" Nov 01 12:56:03.530909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.530857 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" containerName="revoke-keys" Nov 01 12:56:03.530996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.530986 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" containerName="revoke-keys" Nov 01 12:56:03.534821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.534797 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.593074 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.593041 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.593242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.593109 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.593361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.593245 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.593469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.593452 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtw6\" (UniqueName: \"kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.595189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.595123 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:03.633256 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.633228 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0a0610-serving-cert\"" Nov 01 12:56:03.637573 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.637552 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb"] Nov 01 12:56:03.694792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.694712 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtw6\" (UniqueName: \"kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.694934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.694792 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.694934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.694835 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.694934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.694905 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.695441 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:56:03.695410 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-serving-cert: secret "maas-api-e2e-inf-a-0a0610-serving-cert" not found Nov 01 12:56:03.695580 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:56:03.695484 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls podName:3dbc5032-84f7-40fd-876a-0805511d40aa nodeName:}" failed. No retries permitted until 2026-11-01 12:56:04.195462453 +0000 UTC m=+2033.945405429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls") pod "maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" (UID: "3dbc5032-84f7-40fd-876a-0805511d40aa") : secret "maas-api-e2e-inf-a-0a0610-serving-cert" not found Nov 01 12:56:03.695792 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.695764 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.713852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.713828 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.768555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.768523 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtw6\" (UniqueName: \"kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:03.780206 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.780180 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:03.879212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.879175 4484 patch_prober.go:29] interesting pod/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 12:56:03.879407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.879231 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 12:56:03.924528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:03.924502 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:04.188536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.188505 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:04.204716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.204690 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:04.207198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.207173 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:04.269564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.269518 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g"] Nov 01 12:56:04.273000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.272982 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f6b043-bf88g"] Nov 01 12:56:04.375384 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.375349 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e19db5d-9109-4050-a2cc-b91fd5aa848b" path="/var/lib/kubelet/pods/9e19db5d-9109-4050-a2cc-b91fd5aa848b/volumes" Nov 01 12:56:04.466619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.466529 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:04.582617 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.582070 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:04.879331 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.879281 4484 patch_prober.go:29] interesting pod/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 12:56:04.879505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.879363 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 12:56:04.892791 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.892762 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb"] Nov 01 12:56:04.898751 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:56:04.898723 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbc5032_84f7_40fd_876a_0805511d40aa.slice/crio-70ce08a30aaa3b88c6611dddbd25377740cee061a715f8eec66a229a8cd16b53 WatchSource:0}: Error finding container 70ce08a30aaa3b88c6611dddbd25377740cee061a715f8eec66a229a8cd16b53: Status 404 returned error can't find the container with id 70ce08a30aaa3b88c6611dddbd25377740cee061a715f8eec66a229a8cd16b53 Nov 01 12:56:04.900263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:04.900242 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 12:56:05.097936 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:05.097894 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" event={"ID":"3dbc5032-84f7-40fd-876a-0805511d40aa","Type":"ContainerStarted","Data":"70ce08a30aaa3b88c6611dddbd25377740cee061a715f8eec66a229a8cd16b53"} Nov 01 12:56:05.725414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:05.725383 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:05.868321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:05.866079 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:05.879673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:05.879554 4484 patch_prober.go:29] interesting pod/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 12:56:05.879673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:05.879595 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 12:56:06.154625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.153368 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" event={"ID":"3dbc5032-84f7-40fd-876a-0805511d40aa","Type":"ContainerStarted","Data":"59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf"} Nov 01 12:56:06.154625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.154414 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:06.222554 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.222516 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:06.281484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.281417 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" podStartSLOduration=2.40064703 podStartE2EDuration="3.281396487s" totalImagesPullingTime="880.749457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:56:06.280150626 +0000 UTC m=+2036.030093604" watchObservedRunningTime="2026-11-01 12:56:06.281396487 +0000 UTC m=+2036.031339471" Nov 01 12:56:06.523976 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.523876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:06.878495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.878409 4484 patch_prober.go:29] interesting pod/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 12:56:06.878495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:06.878475 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 12:56:07.940346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:07.938324 4484 patch_prober.go:29] interesting pod/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 12:56:07.940346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:07.938387 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 12:56:08.882483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:08.882454 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:56:08.883679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:08.883655 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:56:09.189958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:09.189799 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:09.473729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:09.473657 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:09.520687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:09.520660 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:56:10.784827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:10.784761 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:10.929323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:10.927178 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:11.130428 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:11.130293 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:11.331202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:11.329704 4484 scope.go:122] "RemoveContainer" containerID="3f1dc7253b17e2e2d5b91a1a336027d2ed1d7ae85ae9352c83df15689587285e" Nov 01 12:56:11.525206 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:11.525170 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:12.027798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.027766 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs"] Nov 01 12:56:12.034005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.033982 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.084643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.084616 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs"] Nov 01 12:56:12.126387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.126361 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:56:12.126387 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.126375 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:56:12.186744 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.186719 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:56:12.190761 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.190739 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldw5\" (UniqueName: \"kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.190843 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.190811 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.190990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.190976 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.292460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.292387 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.292626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.292602 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldw5\" (UniqueName: \"kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.292742 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.292724 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.293127 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.293104 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.295104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.295079 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.349086 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.349059 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldw5\" (UniqueName: \"kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:12.648929 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:12.648859 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:13.029635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:13.027229 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs"] Nov 01 12:56:13.314849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:13.314775 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" event={"ID":"d8f64420-b8b9-4f4f-be8e-3b958b025252","Type":"ContainerStarted","Data":"0151229f05219bc6d4b54fb19eeb08f1992124f93cca017e67a8ad539f15c5ff"} Nov 01 12:56:14.323572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:14.323532 4484 generic.go:356] "Generic (PLEG): container finished" podID="d8f64420-b8b9-4f4f-be8e-3b958b025252" containerID="31ad5f05a836573438becd641f3781d0cdefcf9c6aa46e3dd941a07eadeff835" exitCode=0 Nov 01 12:56:14.324014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:14.323609 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" event={"ID":"d8f64420-b8b9-4f4f-be8e-3b958b025252","Type":"ContainerDied","Data":"31ad5f05a836573438becd641f3781d0cdefcf9c6aa46e3dd941a07eadeff835"} Nov 01 12:56:15.282675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.282645 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:15.563281 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.563251 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:15.738365 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.738335 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca\") pod \"d8f64420-b8b9-4f4f-be8e-3b958b025252\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " Nov 01 12:56:15.738515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.738430 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca\") pod \"d8f64420-b8b9-4f4f-be8e-3b958b025252\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " Nov 01 12:56:15.738515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.738456 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5\" (UniqueName: \"kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5\") pod \"d8f64420-b8b9-4f4f-be8e-3b958b025252\" (UID: \"d8f64420-b8b9-4f4f-be8e-3b958b025252\") " Nov 01 12:56:15.738704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.738680 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca" pod "d8f64420-b8b9-4f4f-be8e-3b958b025252" (UID: "d8f64420-b8b9-4f4f-be8e-3b958b025252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:56:15.739048 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.739023 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:15.740515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.740487 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca" pod "d8f64420-b8b9-4f4f-be8e-3b958b025252" (UID: "d8f64420-b8b9-4f4f-be8e-3b958b025252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:56:15.740621 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.740535 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5" pod "d8f64420-b8b9-4f4f-be8e-3b958b025252" (UID: "d8f64420-b8b9-4f4f-be8e-3b958b025252"). InnerVolumeSpecName "kube-api-access-6ldw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:56:15.840495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.840434 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f64420-b8b9-4f4f-be8e-3b958b025252-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:15.840495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:15.840459 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ldw5\" (UniqueName: \"kubernetes.io/projected/d8f64420-b8b9-4f4f-be8e-3b958b025252-kube-api-access-6ldw5\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:16.338243 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:16.338212 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" Nov 01 12:56:16.338425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:16.338243 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs" event={"ID":"d8f64420-b8b9-4f4f-be8e-3b958b025252","Type":"ContainerDied","Data":"0151229f05219bc6d4b54fb19eeb08f1992124f93cca017e67a8ad539f15c5ff"} Nov 01 12:56:16.338425 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:16.338285 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0151229f05219bc6d4b54fb19eeb08f1992124f93cca017e67a8ad539f15c5ff" Nov 01 12:56:21.032391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:21.032356 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:21.153938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:21.153903 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:21.269186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:21.269152 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:23.160696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:23.160666 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:23.291101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:23.290985 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:25.912467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:25.912432 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:26.046186 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:26.046158 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:27.182598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:27.182555 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:30.253051 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:56:30.253014 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:56:33.166114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:33.166084 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:38.140695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:38.140658 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:40.115702 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:40.115667 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:40.335413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:40.335355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:40.510046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:40.509971 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:40.638525 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:40.638491 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:40.706376 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:40.706342 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs"] Nov 01 12:56:40.770111 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:40.770029 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c4bbec-ngxjs"] Nov 01 12:56:41.047349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:41.047252 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:42.372134 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:42.372094 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f64420-b8b9-4f4f-be8e-3b958b025252" path="/var/lib/kubelet/pods/d8f64420-b8b9-4f4f-be8e-3b958b025252/volumes" Nov 01 12:56:44.819351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:44.819311 4484 generic.go:356] "Generic (PLEG): container finished" podID="fd70cefc-cb18-4e20-a779-3f495184858d" containerID="6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21" exitCode=0 Nov 01 12:56:44.819351 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:44.819328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" event={"ID":"fd70cefc-cb18-4e20-a779-3f495184858d","Type":"ContainerDied","Data":"6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21"} Nov 01 12:56:44.819805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:44.819741 4484 scope.go:122] "RemoveContainer" containerID="6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21" Nov 01 12:56:44.821475 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:44.821439 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:56:44.821584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:44.821537 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:56:44.821649 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:44.821588 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:56:45.835267 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:45.835172 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" event={"ID":"fd70cefc-cb18-4e20-a779-3f495184858d","Type":"ContainerStarted","Data":"3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8"} Nov 01 12:56:46.368724 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:46.368642 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:46.642677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:46.642593 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:56:46.642986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:46.642962 4484 patch_prober.go:29] interesting pod/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 01 12:56:46.643165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:46.643138 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 01 12:56:47.376684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:47.376651 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:47.499241 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:47.499210 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:47.660371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:47.645619 4484 patch_prober.go:29] interesting pod/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 01 12:56:47.660371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:47.645688 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 01 12:56:47.704937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:47.704908 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:48.333234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.333207 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:48.657721 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.657631 4484 patch_prober.go:29] interesting pod/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 01 12:56:48.658942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.658909 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 01 12:56:48.913799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.913722 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh"] Nov 01 12:56:48.914445 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.914430 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d8f64420-b8b9-4f4f-be8e-3b958b025252" containerName="revoke-keys" Nov 01 12:56:48.914503 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.914446 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f64420-b8b9-4f4f-be8e-3b958b025252" containerName="revoke-keys" Nov 01 12:56:48.914543 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.914538 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f64420-b8b9-4f4f-be8e-3b958b025252" containerName="revoke-keys" Nov 01 12:56:48.918452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.918433 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:48.976627 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:48.976596 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh"] Nov 01 12:56:49.012667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.012642 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:56:49.012826 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.012703 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:56:49.032047 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.032020 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxh4\" (UniqueName: \"kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.032156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.032137 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.032209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.032160 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.134158 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.134127 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.134158 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.134159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.134433 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.134419 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxh4\" (UniqueName: \"kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.134907 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.134883 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.136575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.136555 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.187820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.187758 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxh4\" (UniqueName: \"kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4\") pod \"maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.232670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.232645 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:49.634967 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.634933 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh"] Nov 01 12:56:49.640803 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:56:49.640759 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0c1f43_a2f2_4c5f_9c0e_42de8e2b2dd2.slice/crio-76640cfba8d0a81e95bdfd8e9b330bc7ede953f7a4653ed15acd2ac70507b0b8 WatchSource:0}: Error finding container 76640cfba8d0a81e95bdfd8e9b330bc7ede953f7a4653ed15acd2ac70507b0b8: Status 404 returned error can't find the container with id 76640cfba8d0a81e95bdfd8e9b330bc7ede953f7a4653ed15acd2ac70507b0b8 Nov 01 12:56:49.666919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.666485 4484 patch_prober.go:29] interesting pod/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 01 12:56:49.666919 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.666541 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 01 12:56:49.997798 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:49.997751 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" event={"ID":"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2","Type":"ContainerStarted","Data":"76640cfba8d0a81e95bdfd8e9b330bc7ede953f7a4653ed15acd2ac70507b0b8"} Nov 01 12:56:50.647369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:50.645634 4484 patch_prober.go:29] interesting pod/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 01 12:56:50.647369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:50.645692 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 01 12:56:51.009955 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:51.009915 4484 generic.go:356] "Generic (PLEG): container finished" podID="af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" containerID="995a5674297bbc04c3d976b04c477962ea03cf1d01573c14bba3385c9f51d4a6" exitCode=0 Nov 01 12:56:51.010464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:51.009999 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" event={"ID":"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2","Type":"ContainerDied","Data":"995a5674297bbc04c3d976b04c477962ea03cf1d01573c14bba3385c9f51d4a6"} Nov 01 12:56:51.647312 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:51.647279 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:56:51.647821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:51.647797 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:56:51.648554 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:51.648532 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:56:52.160249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.160216 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:52.176787 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.176763 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:52.303095 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.303007 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca\") pod \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " Nov 01 12:56:52.303266 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.303164 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4\" (UniqueName: \"kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4\") pod \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " Nov 01 12:56:52.303396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.303262 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca\") pod \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\" (UID: \"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2\") " Nov 01 12:56:52.304657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.304630 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca" pod "af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" (UID: "af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:56:52.307348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.306318 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4" pod "af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" (UID: "af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2"). InnerVolumeSpecName "kube-api-access-hcxh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:56:52.307348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.306632 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca" pod "af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" (UID: "af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:56:52.405219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.405193 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:52.405219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.405221 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:52.405396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:52.405232 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcxh4\" (UniqueName: \"kubernetes.io/projected/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2-kube-api-access-hcxh4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:53.025240 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:53.025210 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" Nov 01 12:56:53.025429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:53.025254 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh" event={"ID":"af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2","Type":"ContainerDied","Data":"76640cfba8d0a81e95bdfd8e9b330bc7ede953f7a4653ed15acd2ac70507b0b8"} Nov 01 12:56:53.025429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:53.025325 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76640cfba8d0a81e95bdfd8e9b330bc7ede953f7a4653ed15acd2ac70507b0b8" Nov 01 12:56:53.306946 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:53.306869 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf"] Nov 01 12:56:53.457651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:53.457613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:54.334710 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.334671 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk"] Nov 01 12:56:54.335667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.335636 4484 cpu_manager.go:414] "containerMap: removing container" podUID="af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" containerName="revoke-keys" Nov 01 12:56:54.335667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.335662 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" containerName="revoke-keys" Nov 01 12:56:54.335924 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.335910 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" containerName="revoke-keys" Nov 01 12:56:54.339616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.339599 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.392520 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.392486 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk"] Nov 01 12:56:54.432948 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.432919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.433131 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.433104 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbhc\" (UniqueName: \"kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.433246 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.433223 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.433524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.433496 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:56:54.433524 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.433508 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:56:54.535188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.535154 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.535350 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.535331 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbhc\" (UniqueName: \"kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.535453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.535439 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.535837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.535814 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.537816 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.537797 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.588123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.588049 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbhc\" (UniqueName: \"kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc\") pod \"maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:54.654021 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:54.653994 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:55.027132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:55.027099 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk"] Nov 01 12:56:55.032295 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:56:55.032252 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6720e72_5808_4a9e_bf48_2102e93d1620.slice/crio-6b80291b076aa7c9d347247d8f186eaca3a2ea6efaa8b6ee6f7ddc5a9f40aeb0 WatchSource:0}: Error finding container 6b80291b076aa7c9d347247d8f186eaca3a2ea6efaa8b6ee6f7ddc5a9f40aeb0: Status 404 returned error can't find the container with id 6b80291b076aa7c9d347247d8f186eaca3a2ea6efaa8b6ee6f7ddc5a9f40aeb0 Nov 01 12:56:56.048150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:56.048120 4484 generic.go:356] "Generic (PLEG): container finished" podID="c6720e72-5808-4a9e-bf48-2102e93d1620" containerID="bb41658818197d2b3520cfc85672a72e1c6e67cd1859dbe2aa87f4ea599062a0" exitCode=0 Nov 01 12:56:56.048555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:56.048212 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" event={"ID":"c6720e72-5808-4a9e-bf48-2102e93d1620","Type":"ContainerDied","Data":"bb41658818197d2b3520cfc85672a72e1c6e67cd1859dbe2aa87f4ea599062a0"} Nov 01 12:56:56.048555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:56.048269 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" event={"ID":"c6720e72-5808-4a9e-bf48-2102e93d1620","Type":"ContainerStarted","Data":"6b80291b076aa7c9d347247d8f186eaca3a2ea6efaa8b6ee6f7ddc5a9f40aeb0"} Nov 01 12:56:57.229316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.229279 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:57.371518 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.371425 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca\") pod \"c6720e72-5808-4a9e-bf48-2102e93d1620\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " Nov 01 12:56:57.371691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.371624 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca\") pod \"c6720e72-5808-4a9e-bf48-2102e93d1620\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " Nov 01 12:56:57.371691 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.371653 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc\" (UniqueName: \"kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc\") pod \"c6720e72-5808-4a9e-bf48-2102e93d1620\" (UID: \"c6720e72-5808-4a9e-bf48-2102e93d1620\") " Nov 01 12:56:57.372064 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.372033 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca" pod "c6720e72-5808-4a9e-bf48-2102e93d1620" (UID: "c6720e72-5808-4a9e-bf48-2102e93d1620"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:56:57.372404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.372370 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:57.373929 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.373897 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc" pod "c6720e72-5808-4a9e-bf48-2102e93d1620" (UID: "c6720e72-5808-4a9e-bf48-2102e93d1620"). InnerVolumeSpecName "kube-api-access-vkbhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:56:57.375364 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.375336 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca" pod "c6720e72-5808-4a9e-bf48-2102e93d1620" (UID: "c6720e72-5808-4a9e-bf48-2102e93d1620"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:56:57.474083 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.474054 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbhc\" (UniqueName: \"kubernetes.io/projected/c6720e72-5808-4a9e-bf48-2102e93d1620-kube-api-access-vkbhc\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:57.474083 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.474079 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720e72-5808-4a9e-bf48-2102e93d1620-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:56:57.907484 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:57.907448 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:56:58.063716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:58.063686 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" Nov 01 12:56:58.063888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:58.063715 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk" event={"ID":"c6720e72-5808-4a9e-bf48-2102e93d1620","Type":"ContainerDied","Data":"6b80291b076aa7c9d347247d8f186eaca3a2ea6efaa8b6ee6f7ddc5a9f40aeb0"} Nov 01 12:56:58.063888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:56:58.063760 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b80291b076aa7c9d347247d8f186eaca3a2ea6efaa8b6ee6f7ddc5a9f40aeb0" Nov 01 12:57:00.257404 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:00.257371 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:57:00.451021 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:00.450987 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m"] Nov 01 12:57:00.555436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:00.555330 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp"] Nov 01 12:57:00.794392 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:00.794361 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:00.943705 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:00.943511 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:03.160538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:03.160500 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:57:03.848536 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:03.848507 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:05.457326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:05.455361 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerName="payload-processing" containerID="cri-o://67e12cef59740c385be2f1179a70ef4013afdef6b7f7eaaf83aaad4046338c00" gracePeriod=25 Nov 01 12:57:05.563325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:05.562711 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerName="payload-pre-processing" containerID="cri-o://85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb" gracePeriod=25 Nov 01 12:57:05.869207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:05.868214 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5"] Nov 01 12:57:05.972038 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:05.972000 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh"] Nov 01 12:57:06.198673 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:06.198592 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:06.345416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:06.345381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:07.011646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:07.011613 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7"] Nov 01 12:57:07.012162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:07.011827 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" containerID="cri-o://a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1" gracePeriod=30 Nov 01 12:57:07.015695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:07.015670 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j"] Nov 01 12:57:07.015899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:07.015865 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" containerID="cri-o://030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d" gracePeriod=30 Nov 01 12:57:08.328238 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.328202 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" containerID="cri-o://ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651" gracePeriod=45 Nov 01 12:57:08.678565 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.678539 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:57:08.775758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.775725 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache\") pod \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " Nov 01 12:57:08.775950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.775876 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm\") pod \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " Nov 01 12:57:08.775950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.775934 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs\") pod \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " Nov 01 12:57:08.776070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.775961 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home\") pod \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " Nov 01 12:57:08.776070 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.775960 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache" pod "5da1edfe-7017-482e-bb96-5aaf5d68bdd5" (UID: "5da1edfe-7017-482e-bb96-5aaf5d68bdd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:08.776175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776069 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir\") pod \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " Nov 01 12:57:08.776175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776138 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s\" (UniqueName: \"kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s\") pod \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\" (UID: \"5da1edfe-7017-482e-bb96-5aaf5d68bdd5\") " Nov 01 12:57:08.776317 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776180 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home" pod "5da1edfe-7017-482e-bb96-5aaf5d68bdd5" (UID: "5da1edfe-7017-482e-bb96-5aaf5d68bdd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:08.776385 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776358 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir" pod "5da1edfe-7017-482e-bb96-5aaf5d68bdd5" (UID: "5da1edfe-7017-482e-bb96-5aaf5d68bdd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:08.776736 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776716 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:08.776845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776738 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:08.776845 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.776751 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:08.778215 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.778178 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm" pod "5da1edfe-7017-482e-bb96-5aaf5d68bdd5" (UID: "5da1edfe-7017-482e-bb96-5aaf5d68bdd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:08.778381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.778228 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs" pod "5da1edfe-7017-482e-bb96-5aaf5d68bdd5" (UID: "5da1edfe-7017-482e-bb96-5aaf5d68bdd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:57:08.778381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.778318 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s" pod "5da1edfe-7017-482e-bb96-5aaf5d68bdd5" (UID: "5da1edfe-7017-482e-bb96-5aaf5d68bdd5"). InnerVolumeSpecName "kube-api-access-92q5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:08.879178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.879141 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:08.879178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.879176 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:08.879495 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:08.879193 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92q5s\" (UniqueName: \"kubernetes.io/projected/5da1edfe-7017-482e-bb96-5aaf5d68bdd5-kube-api-access-92q5s\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:09.063479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.054760 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:09.282324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.275746 4484 generic.go:356] "Generic (PLEG): container finished" podID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerID="ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651" exitCode=0 Nov 01 12:57:09.282324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.275845 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" event={"ID":"5da1edfe-7017-482e-bb96-5aaf5d68bdd5","Type":"ContainerDied","Data":"ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651"} Nov 01 12:57:09.282324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.275882 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" event={"ID":"5da1edfe-7017-482e-bb96-5aaf5d68bdd5","Type":"ContainerDied","Data":"cc7e950d47a9490ade1e7378329269ec05cd74461de8af9bcee711b24b5380c5"} Nov 01 12:57:09.282324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.275906 4484 scope.go:122] "RemoveContainer" containerID="ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651" Nov 01 12:57:09.282324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.276081 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf" Nov 01 12:57:09.412804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.404738 4484 scope.go:122] "RemoveContainer" containerID="ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651" Nov 01 12:57:09.412804 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:09.405182 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651\": container with ID starting with ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651 not found: ID does not exist" containerID="ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651" Nov 01 12:57:09.412804 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.405215 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651"} err="failed to get container status \"ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651\": rpc error: code = NotFound desc = could not find container \"ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651\": container with ID starting with ab352bb6c978fa1cb3ce7cb9b1cef1c4c168b8bcf6bff6fa3af609f77e243651 not found: ID does not exist" Nov 01 12:57:09.619423 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.619345 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf"] Nov 01 12:57:09.622161 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:09.622143 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-2-w2-c7206f-kserve-84f878b8-7gfqf"] Nov 01 12:57:10.373010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:10.372981 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" path="/var/lib/kubelet/pods/5da1edfe-7017-482e-bb96-5aaf5d68bdd5/volumes" Nov 01 12:57:10.839193 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:10.839155 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:10.869206 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:10.869172 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" podUID="237c99e9-2d6b-43e4-8858-8361d1fcc4f6" containerName="payload-processing" containerID="cri-o://7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b" gracePeriod=25 Nov 01 12:57:10.973147 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:10.973114 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" podUID="56fbf03b-f3e9-4994-9248-8f30e963748a" containerName="payload-pre-processing" containerID="cri-o://b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249" gracePeriod=25 Nov 01 12:57:11.159672 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.159633 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:57:11.229680 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.229644 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg\" (UniqueName: \"kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg\") pod \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " Nov 01 12:57:11.229850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.229831 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume\") pod \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\" (UID: \"237c99e9-2d6b-43e4-8858-8361d1fcc4f6\") " Nov 01 12:57:11.230229 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.230201 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume" pod "237c99e9-2d6b-43e4-8858-8361d1fcc4f6" (UID: "237c99e9-2d6b-43e4-8858-8361d1fcc4f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:11.230576 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.230556 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:11.232004 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.231983 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg" pod "237c99e9-2d6b-43e4-8858-8361d1fcc4f6" (UID: "237c99e9-2d6b-43e4-8858-8361d1fcc4f6"). InnerVolumeSpecName "kube-api-access-kkpqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:11.261604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.261583 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:57:11.298885 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.298852 4484 generic.go:356] "Generic (PLEG): container finished" podID="56fbf03b-f3e9-4994-9248-8f30e963748a" containerID="b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249" exitCode=0 Nov 01 12:57:11.299076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.298990 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" event={"ID":"56fbf03b-f3e9-4994-9248-8f30e963748a","Type":"ContainerDied","Data":"b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249"} Nov 01 12:57:11.299076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.299035 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" event={"ID":"56fbf03b-f3e9-4994-9248-8f30e963748a","Type":"ContainerDied","Data":"7cc53bdc2cbf7b1eb7ea2048b125c52eb0701a8d5ee04a4f145ac4de12447ce1"} Nov 01 12:57:11.299076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.299057 4484 scope.go:122] "RemoveContainer" containerID="b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249" Nov 01 12:57:11.299257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.299168 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh" Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.332143 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8\" (UniqueName: \"kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8\") pod \"56fbf03b-f3e9-4994-9248-8f30e963748a\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.332212 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume\") pod \"56fbf03b-f3e9-4994-9248-8f30e963748a\" (UID: \"56fbf03b-f3e9-4994-9248-8f30e963748a\") " Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.333382 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpqg\" (UniqueName: \"kubernetes.io/projected/237c99e9-2d6b-43e4-8858-8361d1fcc4f6-kube-api-access-kkpqg\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.334482 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume" pod "56fbf03b-f3e9-4994-9248-8f30e963748a" (UID: "56fbf03b-f3e9-4994-9248-8f30e963748a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.334936 4484 generic.go:356] "Generic (PLEG): container finished" podID="237c99e9-2d6b-43e4-8858-8361d1fcc4f6" containerID="7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b" exitCode=0 Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.335055 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" event={"ID":"237c99e9-2d6b-43e4-8858-8361d1fcc4f6","Type":"ContainerDied","Data":"7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b"} Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.335108 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" event={"ID":"237c99e9-2d6b-43e4-8858-8361d1fcc4f6","Type":"ContainerDied","Data":"9fa333b79d622f7d99883272e00b6c32c9e646e1154c071852e4534cc18ad613"} Nov 01 12:57:11.344491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.335188 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5" Nov 01 12:57:11.395930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.367716 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 12:57:11.395930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.371999 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 12:57:11.400797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.400764 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8" pod "56fbf03b-f3e9-4994-9248-8f30e963748a" (UID: "56fbf03b-f3e9-4994-9248-8f30e963748a"). InnerVolumeSpecName "kube-api-access-cdjj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:11.409819 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.409786 4484 scope.go:122] "RemoveContainer" containerID="b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249" Nov 01 12:57:11.410157 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:11.410131 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249\": container with ID starting with b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249 not found: ID does not exist" containerID="b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249" Nov 01 12:57:11.410261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.410164 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249"} err="failed to get container status \"b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249\": rpc error: code = NotFound desc = could not find container \"b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249\": container with ID starting with b8b4c0d06f915a5fa6bb0a6fdd132f8bf9ec7a34a7644f7e2198922461c0b249 not found: ID does not exist" Nov 01 12:57:11.410261 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.410183 4484 scope.go:122] "RemoveContainer" containerID="7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b" Nov 01 12:57:11.427618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.427570 4484 scope.go:122] "RemoveContainer" containerID="7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b" Nov 01 12:57:11.427917 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:11.427886 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b\": container with ID starting with 7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b not found: ID does not exist" containerID="7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b" Nov 01 12:57:11.428039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.427927 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b"} err="failed to get container status \"7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b\": rpc error: code = NotFound desc = could not find container \"7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b\": container with ID starting with 7f86ca7f6d7864dbaea0892de7065ca51f129321aee052fb6028ad31cf38400b not found: ID does not exist" Nov 01 12:57:11.435056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.435024 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdjj8\" (UniqueName: \"kubernetes.io/projected/56fbf03b-f3e9-4994-9248-8f30e963748a-kube-api-access-cdjj8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:11.435172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.435061 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fbf03b-f3e9-4994-9248-8f30e963748a-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:11.446679 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.446654 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk"] Nov 01 12:57:11.452675 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.452650 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6b9030-6bsbk"] Nov 01 12:57:11.517239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.517201 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5"] Nov 01 12:57:11.523704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.523672 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6b9030-86dcb5cf5f-9dwh5"] Nov 01 12:57:11.683007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.682900 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-dkb6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:57:11.683007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.682982 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:57:11.715326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.715218 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh"] Nov 01 12:57:11.721346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:11.721318 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6b9030-54f7f4b4d7-8hlgh"] Nov 01 12:57:12.201989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.198630 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:12.339219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.337974 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:57:12.340967 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.340944 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:57:12.343502 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.343478 4484 generic.go:356] "Generic (PLEG): container finished" podID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerID="a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1" exitCode=0 Nov 01 12:57:12.343625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.343510 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" event={"ID":"1bea2e7e-a9c6-46b0-b930-8c769d293f66","Type":"ContainerDied","Data":"a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1"} Nov 01 12:57:12.343625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.343520 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" Nov 01 12:57:12.343625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.343605 4484 scope.go:122] "RemoveContainer" containerID="a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1" Nov 01 12:57:12.343802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.343595 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7" event={"ID":"1bea2e7e-a9c6-46b0-b930-8c769d293f66","Type":"ContainerDied","Data":"450f25c3245e0e2188625a9b0123c49a9a8187153d1cd547b9d948daa813bc5f"} Nov 01 12:57:12.346252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.346230 4484 generic.go:356] "Generic (PLEG): container finished" podID="106511c1-d902-40c3-a306-a5b8790400e4" containerID="030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d" exitCode=0 Nov 01 12:57:12.346367 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.346277 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" Nov 01 12:57:12.346438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.346370 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" event={"ID":"106511c1-d902-40c3-a306-a5b8790400e4","Type":"ContainerDied","Data":"030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d"} Nov 01 12:57:12.346438 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.346417 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j" event={"ID":"106511c1-d902-40c3-a306-a5b8790400e4","Type":"ContainerDied","Data":"4d99ad21b44ad895acfbcd9b6fb624cd45668053c716543b6a1f7530506ee45f"} Nov 01 12:57:12.356114 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.356046 4484 scope.go:122] "RemoveContainer" containerID="a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1" Nov 01 12:57:12.356522 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:12.356407 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1\": container with ID starting with a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1 not found: ID does not exist" containerID="a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1" Nov 01 12:57:12.356522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.356438 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1"} err="failed to get container status \"a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1\": rpc error: code = NotFound desc = could not find container \"a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1\": container with ID starting with a907e16bd194bc90a457238162badc9575b509623bcbffa45b38bc623db16ae1 not found: ID does not exist" Nov 01 12:57:12.356522 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.356459 4484 scope.go:122] "RemoveContainer" containerID="030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d" Nov 01 12:57:12.367058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.367044 4484 scope.go:122] "RemoveContainer" containerID="030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d" Nov 01 12:57:12.367370 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:12.367348 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d\": container with ID starting with 030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d not found: ID does not exist" containerID="030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d" Nov 01 12:57:12.367463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.367373 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d"} err="failed to get container status \"030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d\": rpc error: code = NotFound desc = could not find container \"030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d\": container with ID starting with 030f8b56520806e89c84d8bdeb52d18e31cec95be2949505b19c7d08d4ed8c9d not found: ID does not exist" Nov 01 12:57:12.370866 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.370846 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237c99e9-2d6b-43e4-8858-8361d1fcc4f6" path="/var/lib/kubelet/pods/237c99e9-2d6b-43e4-8858-8361d1fcc4f6/volumes" Nov 01 12:57:12.371246 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.371233 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fbf03b-f3e9-4994-9248-8f30e963748a" path="/var/lib/kubelet/pods/56fbf03b-f3e9-4994-9248-8f30e963748a/volumes" Nov 01 12:57:12.371693 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.371678 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6720e72-5808-4a9e-bf48-2102e93d1620" path="/var/lib/kubelet/pods/c6720e72-5808-4a9e-bf48-2102e93d1620/volumes" Nov 01 12:57:12.453082 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453045 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.453242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453099 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.453242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453168 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.453242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453236 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.453416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453278 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.453416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453356 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.453416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453408 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.453646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453449 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.453646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453486 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.453646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453517 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.453860 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.453657 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454460 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454592 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454648 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454692 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454696 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454710 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454729 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy\") pod \"106511c1-d902-40c3-a306-a5b8790400e4\" (UID: \"106511c1-d902-40c3-a306-a5b8790400e4\") " Nov 01 12:57:12.455180 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.454856 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.455725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.455202 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.455725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.455533 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:12.455725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.455586 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.455725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.455623 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.455725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.455675 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8\") pod \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\" (UID: \"1bea2e7e-a9c6-46b0-b930-8c769d293f66\") " Nov 01 12:57:12.456498 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456438 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106511c1-d902-40c3-a306-a5b8790400e4-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.456682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456462 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.456682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456550 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.456682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456563 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.456682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456574 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.456682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456586 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.456965 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.456748 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.457239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457189 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:12.457487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457430 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.457487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457447 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.457644 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457545 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "kube-api-access-ccchn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:12.457888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457831 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:12.457888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457857 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.457998 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.457889 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo" pod "106511c1-d902-40c3-a306-a5b8790400e4" (UID: "106511c1-d902-40c3-a306-a5b8790400e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:12.459226 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.459204 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:12.459352 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.459231 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:12.459626 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.459603 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:12.460587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.460568 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8" pod "1bea2e7e-a9c6-46b0-b930-8c769d293f66" (UID: "1bea2e7e-a9c6-46b0-b930-8c769d293f66"). InnerVolumeSpecName "kube-api-access-g4jl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:12.558779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558705 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.558779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558738 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.558779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558754 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106511c1-d902-40c3-a306-a5b8790400e4-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.558779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558768 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.558779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558782 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558797 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jl8\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-kube-api-access-g4jl8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558814 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccchn\" (UniqueName: \"kubernetes.io/projected/106511c1-d902-40c3-a306-a5b8790400e4-kube-api-access-ccchn\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558828 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bea2e7e-a9c6-46b0-b930-8c769d293f66-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558841 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106511c1-d902-40c3-a306-a5b8790400e4-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558855 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558871 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.559162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.558884 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bea2e7e-a9c6-46b0-b930-8c769d293f66-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:12.765150 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.765122 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7"] Nov 01 12:57:12.770914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.770881 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gdbv7"] Nov 01 12:57:12.870881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.870795 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j"] Nov 01 12:57:12.876136 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:12.876112 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-dkb6j"] Nov 01 12:57:13.384501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:13.384461 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 12:57:13.384859 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:13.384522 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 12:57:14.366783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.366708 4484 generic.go:356] "Generic (PLEG): container finished" podID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerID="2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7" exitCode=0 Nov 01 12:57:14.370964 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.370942 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106511c1-d902-40c3-a306-a5b8790400e4" path="/var/lib/kubelet/pods/106511c1-d902-40c3-a306-a5b8790400e4/volumes" Nov 01 12:57:14.371448 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.371433 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" path="/var/lib/kubelet/pods/1bea2e7e-a9c6-46b0-b930-8c769d293f66/volumes" Nov 01 12:57:14.371758 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.371741 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" event={"ID":"4fc4d29d-9be9-42e5-a95a-59df4aa746f0","Type":"ContainerDied","Data":"2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7"} Nov 01 12:57:14.372058 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.372043 4484 scope.go:122] "RemoveContainer" containerID="2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7" Nov 01 12:57:14.373465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.373434 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:57:14.373587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.373518 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:57:14.373587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:14.373567 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:57:15.376480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:15.376444 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" event={"ID":"4fc4d29d-9be9-42e5-a95a-59df4aa746f0","Type":"ContainerStarted","Data":"fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b"} Nov 01 12:57:15.562061 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:15.562021 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:57:15.562463 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:15.562425 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 12:57:15.562640 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:15.562609 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 12:57:16.566326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:16.566007 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 12:57:16.566326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:16.566062 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 12:57:17.168429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:17.168381 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:17.551686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:17.551645 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 12:57:17.551941 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:17.551908 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 12:57:18.137259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.137227 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:18.279325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.279286 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:18.473820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.473712 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:18.475597 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.475572 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f"] Nov 01 12:57:18.475787 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.475764 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" containerID="cri-o://8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64" gracePeriod=30 Nov 01 12:57:18.555173 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.555134 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 12:57:18.559455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:18.559413 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 12:57:19.566437 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:19.566396 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 12:57:19.566856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:19.566472 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 12:57:20.565436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:20.565401 4484 patch_prober.go:29] interesting pod/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 12:57:20.565745 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:20.565701 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 12:57:20.678188 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:20.678150 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:21.321394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.321337 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:21.359645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.359604 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 12:57:21.361103 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.361071 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 12:57:21.555458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.555431 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:57:21.555893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.555869 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:57:21.556453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.556435 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 12:57:21.711431 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:21.711279 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:22.070986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.070951 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr"] Nov 01 12:57:22.071176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.071154 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" containerID="cri-o://cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04" gracePeriod=30 Nov 01 12:57:22.104562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.104530 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj"] Nov 01 12:57:22.104809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.104774 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" containerID="cri-o://a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a" gracePeriod=30 Nov 01 12:57:22.108416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.108390 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj"] Nov 01 12:57:22.108584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.108564 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" containerID="cri-o://7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942" gracePeriod=30 Nov 01 12:57:22.329793 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.329708 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6"] Nov 01 12:57:22.329966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:22.329916 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" containerID="cri-o://8eaeb03574ab76cefe8898c9851db0d62bfebdc7694c3c1cfebb6b506d5e5c93" gracePeriod=30 Nov 01 12:57:23.265179 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.265142 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:23.758926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.758900 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:57:23.883209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883138 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883209 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883201 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883340 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883386 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883477 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883464 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883581 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883565 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883677 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:23.883839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883758 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883794 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.883839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883822 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data\") pod \"11835b7f-717d-48fa-8971-65b79dfa49d9\" (UID: \"11835b7f-717d-48fa-8971-65b79dfa49d9\") " Nov 01 12:57:23.884003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883873 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:23.884003 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.883888 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:23.884100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884066 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:23.884212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884189 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:23.884538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884512 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11835b7f-717d-48fa-8971-65b79dfa49d9-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.884538 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884544 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.884713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884561 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.884713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884576 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.884713 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.884590 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.885548 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.885521 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:23.885683 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.885616 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:23.885895 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.885863 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "kube-api-access-zrjlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:23.892707 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.892685 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy" pod "11835b7f-717d-48fa-8971-65b79dfa49d9" (UID: "11835b7f-717d-48fa-8971-65b79dfa49d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:23.985738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.985709 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.985738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.985737 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.985918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.985747 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjlk\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-kube-api-access-zrjlk\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:23.985918 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:23.985755 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11835b7f-717d-48fa-8971-65b79dfa49d9-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:24.089882 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.089846 4484 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:57:24.090051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.089902 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:57:24.297952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.297916 4484 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:57:24.298418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.297977 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:57:24.478661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.478626 4484 generic.go:356] "Generic (PLEG): container finished" podID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerID="8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64" exitCode=0 Nov 01 12:57:24.478818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.478684 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" Nov 01 12:57:24.478818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.478700 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" event={"ID":"11835b7f-717d-48fa-8971-65b79dfa49d9","Type":"ContainerDied","Data":"8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64"} Nov 01 12:57:24.478818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.478772 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f" event={"ID":"11835b7f-717d-48fa-8971-65b79dfa49d9","Type":"ContainerDied","Data":"770df299a50d8acc1d59c44f431d516a880f80ef3b26f4ca896dd6a809f7185a"} Nov 01 12:57:24.478818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.478801 4484 scope.go:122] "RemoveContainer" containerID="8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64" Nov 01 12:57:24.489383 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.489365 4484 scope.go:122] "RemoveContainer" containerID="0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107" Nov 01 12:57:24.496321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.496291 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k"] Nov 01 12:57:24.496527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.496494 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" containerID="cri-o://9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad" gracePeriod=30 Nov 01 12:57:24.498641 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.498621 4484 scope.go:122] "RemoveContainer" containerID="8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64" Nov 01 12:57:24.498987 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:24.498962 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64\": container with ID starting with 8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64 not found: ID does not exist" containerID="8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64" Nov 01 12:57:24.499067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.498995 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64"} err="failed to get container status \"8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64\": rpc error: code = NotFound desc = could not find container \"8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64\": container with ID starting with 8ce3f1eb771e83b9df3aa813974b83813a49957f0160efcd3014e77df0f5db64 not found: ID does not exist" Nov 01 12:57:24.499067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.499017 4484 scope.go:122] "RemoveContainer" containerID="0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107" Nov 01 12:57:24.499372 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:24.499345 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107\": container with ID starting with 0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107 not found: ID does not exist" containerID="0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107" Nov 01 12:57:24.499462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.499375 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107"} err="failed to get container status \"0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107\": rpc error: code = NotFound desc = could not find container \"0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107\": container with ID starting with 0c6547a9f7df5ddf0d2daaa5a1b6bf1698c2d6333e6881b159678f3ec8700107 not found: ID does not exist" Nov 01 12:57:24.751542 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.751504 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f"] Nov 01 12:57:24.756389 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:24.756364 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6b9030-openshift-default-76bfb4bf6f-58l6f"] Nov 01 12:57:25.489434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.489409 4484 generic.go:356] "Generic (PLEG): container finished" podID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerID="67e12cef59740c385be2f1179a70ef4013afdef6b7f7eaaf83aaad4046338c00" exitCode=0 Nov 01 12:57:25.489761 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.489481 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" event={"ID":"8e820e6b-7262-4c30-acec-14251eaf26ed","Type":"ContainerDied","Data":"67e12cef59740c385be2f1179a70ef4013afdef6b7f7eaaf83aaad4046338c00"} Nov 01 12:57:25.700586 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.700563 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:57:25.812814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.812782 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8\" (UniqueName: \"kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8\") pod \"8e820e6b-7262-4c30-acec-14251eaf26ed\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " Nov 01 12:57:25.812966 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.812922 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume\") pod \"8e820e6b-7262-4c30-acec-14251eaf26ed\" (UID: \"8e820e6b-7262-4c30-acec-14251eaf26ed\") " Nov 01 12:57:25.813949 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.813923 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume" pod "8e820e6b-7262-4c30-acec-14251eaf26ed" (UID: "8e820e6b-7262-4c30-acec-14251eaf26ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:25.815441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.815420 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:57:25.815829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.815807 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8" pod "8e820e6b-7262-4c30-acec-14251eaf26ed" (UID: "8e820e6b-7262-4c30-acec-14251eaf26ed"). InnerVolumeSpecName "kube-api-access-94wb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:25.915005 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.914977 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume\") pod \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " Nov 01 12:57:25.915184 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.915020 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p\" (UniqueName: \"kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p\") pod \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\" (UID: \"e3708e19-6de2-4ba2-a1c7-23e010ab81da\") " Nov 01 12:57:25.915441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.915413 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume" pod "e3708e19-6de2-4ba2-a1c7-23e010ab81da" (UID: "e3708e19-6de2-4ba2-a1c7-23e010ab81da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:25.915544 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.915471 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wb8\" (UniqueName: \"kubernetes.io/projected/8e820e6b-7262-4c30-acec-14251eaf26ed-kube-api-access-94wb8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:25.915544 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.915501 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e820e6b-7262-4c30-acec-14251eaf26ed-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:25.917137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:25.917117 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p" pod "e3708e19-6de2-4ba2-a1c7-23e010ab81da" (UID: "e3708e19-6de2-4ba2-a1c7-23e010ab81da"). InnerVolumeSpecName "kube-api-access-dvp4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:26.016763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.016690 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3708e19-6de2-4ba2-a1c7-23e010ab81da-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:26.016763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.016718 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvp4p\" (UniqueName: \"kubernetes.io/projected/e3708e19-6de2-4ba2-a1c7-23e010ab81da-kube-api-access-dvp4p\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:26.229677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.229638 4484 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:57:26.229839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.229695 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:57:26.260669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.260642 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:26.370487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.370414 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" path="/var/lib/kubelet/pods/11835b7f-717d-48fa-8971-65b79dfa49d9/volumes" Nov 01 12:57:26.423051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.423016 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:26.507807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.507775 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" Nov 01 12:57:26.508198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.507803 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m" event={"ID":"8e820e6b-7262-4c30-acec-14251eaf26ed","Type":"ContainerDied","Data":"dcea8066255c47c719c40dd4f76c90e490b2a92af08f82f0a96a06f5ab322157"} Nov 01 12:57:26.508198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.507876 4484 scope.go:122] "RemoveContainer" containerID="67e12cef59740c385be2f1179a70ef4013afdef6b7f7eaaf83aaad4046338c00" Nov 01 12:57:26.509482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.509456 4484 generic.go:356] "Generic (PLEG): container finished" podID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerID="85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb" exitCode=0 Nov 01 12:57:26.509583 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.509492 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" event={"ID":"e3708e19-6de2-4ba2-a1c7-23e010ab81da","Type":"ContainerDied","Data":"85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb"} Nov 01 12:57:26.509583 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.509520 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" event={"ID":"e3708e19-6de2-4ba2-a1c7-23e010ab81da","Type":"ContainerDied","Data":"2c6e1677bd897f1eb8236d35cd0602bd11298a5df117937fbbba79041dfc6534"} Nov 01 12:57:26.509583 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.509523 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp" Nov 01 12:57:26.517857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.517838 4484 scope.go:122] "RemoveContainer" containerID="85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb" Nov 01 12:57:26.526682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.526660 4484 scope.go:122] "RemoveContainer" containerID="85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb" Nov 01 12:57:26.526939 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:26.526920 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb\": container with ID starting with 85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb not found: ID does not exist" containerID="85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb" Nov 01 12:57:26.526993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.526953 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb"} err="failed to get container status \"85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb\": rpc error: code = NotFound desc = could not find container \"85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb\": container with ID starting with 85bf24fbc52737857109eafe751931b444c922248e75411ecbc90ad25f869efb not found: ID does not exist" Nov 01 12:57:26.670275 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.670208 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp"] Nov 01 12:57:26.715750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.715720 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c7206f-558445b695-rfqlp"] Nov 01 12:57:26.774512 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.774485 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m"] Nov 01 12:57:26.777856 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:26.777831 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c7206f-d7fff7f9f-twt4m"] Nov 01 12:57:27.459795 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.459773 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:57:27.462650 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.462630 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:57:27.470521 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.470489 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:57:27.528436 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.528409 4484 generic.go:356] "Generic (PLEG): container finished" podID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerID="8eaeb03574ab76cefe8898c9851db0d62bfebdc7694c3c1cfebb6b506d5e5c93" exitCode=0 Nov 01 12:57:27.528870 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.528584 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" event={"ID":"76f659fa-cd16-4857-a348-3ed5d08c2714","Type":"ContainerDied","Data":"8eaeb03574ab76cefe8898c9851db0d62bfebdc7694c3c1cfebb6b506d5e5c93"} Nov 01 12:57:27.529975 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.529950 4484 generic.go:356] "Generic (PLEG): container finished" podID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerID="7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942" exitCode=0 Nov 01 12:57:27.530069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.530000 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" Nov 01 12:57:27.530069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.530024 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" event={"ID":"a0a915fc-c906-43b1-8b47-cf0b54d97404","Type":"ContainerDied","Data":"7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942"} Nov 01 12:57:27.530227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.530074 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj" event={"ID":"a0a915fc-c906-43b1-8b47-cf0b54d97404","Type":"ContainerDied","Data":"e0769fad77ba73aba269e21d1a1240b57acf9be91bb7ec2664829bcc043d7cda"} Nov 01 12:57:27.530227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.530108 4484 scope.go:122] "RemoveContainer" containerID="7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942" Nov 01 12:57:27.532805 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.532784 4484 generic.go:356] "Generic (PLEG): container finished" podID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerID="a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a" exitCode=0 Nov 01 12:57:27.532905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.532866 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" event={"ID":"e410b77d-0691-47ef-a10d-b4cf73d4ce56","Type":"ContainerDied","Data":"a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a"} Nov 01 12:57:27.532905 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.532900 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" event={"ID":"e410b77d-0691-47ef-a10d-b4cf73d4ce56","Type":"ContainerDied","Data":"21270cb2a2746d2bcce106a04f5e4939daf71b806a69239714bae7527796a132"} Nov 01 12:57:27.533019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.532964 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj" Nov 01 12:57:27.535474 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535455 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.535587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535504 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.535587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535548 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.535708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535588 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.535708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535628 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.535708 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535694 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.535851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535722 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.535851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535776 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.535851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535831 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535864 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.536006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535931 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.536006 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.535969 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536015 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.536159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536047 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536088 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536115 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.536370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536189 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.536370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536248 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.536370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536277 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536288 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536380 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536430 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536472 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536524 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536565 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536586 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.536636 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536629 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket\") pod \"83b14464-64a5-4ce3-8b8f-65504678c4c8\" (UID: \"83b14464-64a5-4ce3-8b8f-65504678c4c8\") " Nov 01 12:57:27.537025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536671 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket\") pod \"a0a915fc-c906-43b1-8b47-cf0b54d97404\" (UID: \"a0a915fc-c906-43b1-8b47-cf0b54d97404\") " Nov 01 12:57:27.537025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.536756 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token\") pod \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\" (UID: \"e410b77d-0691-47ef-a10d-b4cf73d4ce56\") " Nov 01 12:57:27.537443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.537420 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.537566 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.537449 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.537912 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.537887 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "kube-api-access-z66wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.538144 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.538123 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.538462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.538436 4484 generic.go:356] "Generic (PLEG): container finished" podID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerID="cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04" exitCode=0 Nov 01 12:57:27.538553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.538518 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" event={"ID":"83b14464-64a5-4ce3-8b8f-65504678c4c8","Type":"ContainerDied","Data":"cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04"} Nov 01 12:57:27.538625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.538559 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" event={"ID":"83b14464-64a5-4ce3-8b8f-65504678c4c8","Type":"ContainerDied","Data":"225e230b2a25f609b20a19fb6960028afe215224654db2b3d0be2b3ce28bdb04"} Nov 01 12:57:27.538685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.538639 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr" Nov 01 12:57:27.539999 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.539973 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:27.540196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.540180 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.542019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.541467 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:27.542019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.541795 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542049 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542465 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542667 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542798 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.542900 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.543445 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.543461 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.543731 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.544031 4484 scope.go:122] "RemoveContainer" containerID="7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:27.544400 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942\": container with ID starting with 7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942 not found: ID does not exist" containerID="7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.544432 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942"} err="failed to get container status \"7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942\": rpc error: code = NotFound desc = could not find container \"7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942\": container with ID starting with 7f059d9e65a03513e4e2349a60a23a8a7872ddf64f7d6f285694e4582cb21942 not found: ID does not exist" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.544458 4484 scope.go:122] "RemoveContainer" containerID="a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.546527 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.549497 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.551581 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.551624 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "kube-api-access-7mcsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.551757 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "kube-api-access-j4k8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.553946 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.553957 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo" pod "a0a915fc-c906-43b1-8b47-cf0b54d97404" (UID: "a0a915fc-c906-43b1-8b47-cf0b54d97404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.553977 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token" pod "83b14464-64a5-4ce3-8b8f-65504678c4c8" (UID: "83b14464-64a5-4ce3-8b8f-65504678c4c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.557493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.554002 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token" pod "e410b77d-0691-47ef-a10d-b4cf73d4ce56" (UID: "e410b77d-0691-47ef-a10d-b4cf73d4ce56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.579469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.579451 4484 scope.go:122] "RemoveContainer" containerID="a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a" Nov 01 12:57:27.579773 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:27.579748 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a\": container with ID starting with a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a not found: ID does not exist" containerID="a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a" Nov 01 12:57:27.579830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.579779 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a"} err="failed to get container status \"a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a\": rpc error: code = NotFound desc = could not find container \"a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a\": container with ID starting with a31c4a8f522b1a60646c57c688fb83beabdb212279065470c6c17e9fc0213f2a not found: ID does not exist" Nov 01 12:57:27.579830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.579797 4484 scope.go:122] "RemoveContainer" containerID="cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04" Nov 01 12:57:27.593582 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.593499 4484 scope.go:122] "RemoveContainer" containerID="911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e" Nov 01 12:57:27.625757 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.625741 4484 scope.go:122] "RemoveContainer" containerID="cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04" Nov 01 12:57:27.626031 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:27.626008 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04\": container with ID starting with cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04 not found: ID does not exist" containerID="cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04" Nov 01 12:57:27.626132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.626035 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04"} err="failed to get container status \"cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04\": rpc error: code = NotFound desc = could not find container \"cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04\": container with ID starting with cc437db88a92829613ee3f79c78f83531312f8dace8addc277876fb6a2733f04 not found: ID does not exist" Nov 01 12:57:27.626132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.626052 4484 scope.go:122] "RemoveContainer" containerID="911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e" Nov 01 12:57:27.626365 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:27.626341 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e\": container with ID starting with 911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e not found: ID does not exist" containerID="911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e" Nov 01 12:57:27.626505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.626375 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e"} err="failed to get container status \"911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e\": rpc error: code = NotFound desc = could not find container \"911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e\": container with ID starting with 911bb820800286e6ed36268cd442cb7c2e21497180531b4b8de38774ef57172e not found: ID does not exist" Nov 01 12:57:27.629176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.629158 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:57:27.638373 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638350 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638374 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4k8k\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-kube-api-access-j4k8k\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638386 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638398 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638411 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638420 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638429 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638437 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638446 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mcsv\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-kube-api-access-7mcsv\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638458 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638472 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638480 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638489 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638496 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638504 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b14464-64a5-4ce3-8b8f-65504678c4c8-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638512 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a915fc-c906-43b1-8b47-cf0b54d97404-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638519 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638527 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b14464-64a5-4ce3-8b8f-65504678c4c8-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638536 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a915fc-c906-43b1-8b47-cf0b54d97404-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638546 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638554 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66wc\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-kube-api-access-z66wc\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638562 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a915fc-c906-43b1-8b47-cf0b54d97404-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638570 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638578 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e410b77d-0691-47ef-a10d-b4cf73d4ce56-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.638690 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.638585 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b14464-64a5-4ce3-8b8f-65504678c4c8-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.740157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740123 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740171 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740249 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740296 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740471 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740528 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740558 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740581 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740643 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs\") pod \"76f659fa-cd16-4857-a348-3ed5d08c2714\" (UID: \"76f659fa-cd16-4857-a348-3ed5d08c2714\") " Nov 01 12:57:27.740861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740681 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.740861 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.740802 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:27.741017 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.741000 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.741239 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.741283 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f659fa-cd16-4857-a348-3ed5d08c2714-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.741328 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.743543 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.743707 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.743783 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.744085 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:27.741292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.746233 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "kube-api-access-v9sdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:27.754295 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.754263 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo" pod "76f659fa-cd16-4857-a348-3ed5d08c2714" (UID: "76f659fa-cd16-4857-a348-3ed5d08c2714"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:27.842359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.842333 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.842359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.842357 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.842506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.842367 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9sdp\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-kube-api-access-v9sdp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.842506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.842376 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.842506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.842385 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:27.842506 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:27.842392 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f659fa-cd16-4857-a348-3ed5d08c2714-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:28.017388 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.017352 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj"] Nov 01 12:57:28.023624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.023595 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-xq2bj"] Nov 01 12:57:28.139138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.139058 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr"] Nov 01 12:57:28.144459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.144429 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-r8fcr"] Nov 01 12:57:28.374496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.372492 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" path="/var/lib/kubelet/pods/83b14464-64a5-4ce3-8b8f-65504678c4c8/volumes" Nov 01 12:57:28.374496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.373327 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" path="/var/lib/kubelet/pods/8e820e6b-7262-4c30-acec-14251eaf26ed/volumes" Nov 01 12:57:28.374496 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.374210 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" path="/var/lib/kubelet/pods/a0a915fc-c906-43b1-8b47-cf0b54d97404/volumes" Nov 01 12:57:28.376857 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.376813 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" path="/var/lib/kubelet/pods/e3708e19-6de2-4ba2-a1c7-23e010ab81da/volumes" Nov 01 12:57:28.377499 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.377407 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj"] Nov 01 12:57:28.382741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.382718 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-zt8zj"] Nov 01 12:57:28.547415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.547388 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" Nov 01 12:57:28.547827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.547410 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6" event={"ID":"76f659fa-cd16-4857-a348-3ed5d08c2714","Type":"ContainerDied","Data":"3de601aa14d0d59ece7437de0a70003d02e4e3ff2a10a77933cfc47955d4c5bb"} Nov 01 12:57:28.547827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.547471 4484 scope.go:122] "RemoveContainer" containerID="8eaeb03574ab76cefe8898c9851db0d62bfebdc7694c3c1cfebb6b506d5e5c93" Nov 01 12:57:28.714296 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.714265 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6"] Nov 01 12:57:28.720831 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.720795 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-hz2h6"] Nov 01 12:57:28.795674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.795637 4484 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:57:28.795833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.795709 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:57:28.964002 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:28.963967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:29.577774 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.577746 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:29.774772 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.774689 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:29.829392 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.829367 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894049 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894163 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894206 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894255 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894286 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894507 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894614 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894655 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.894937 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.894682 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket\") pod \"fe467197-9c0b-4c10-aa51-6504f92be21a\" (UID: \"fe467197-9c0b-4c10-aa51-6504f92be21a\") " Nov 01 12:57:29.920451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.899768 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:29.920451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.904459 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:29.920451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.904473 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:29.920451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.913648 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:29.934325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.927038 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:29.934325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.927236 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:29.939344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.936215 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "kube-api-access-fd65d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:29.939344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.936369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:29.939344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.936472 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token" pod "fe467197-9c0b-4c10-aa51-6504f92be21a" (UID: "fe467197-9c0b-4c10-aa51-6504f92be21a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:29.996333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996283 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996321 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996332 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996342 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd65d\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-kube-api-access-fd65d\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996354 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe467197-9c0b-4c10-aa51-6504f92be21a-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996364 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996372 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996381 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:29.996562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:29.996389 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe467197-9c0b-4c10-aa51-6504f92be21a-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:30.039375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.039292 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n"] Nov 01 12:57:30.039996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.039981 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" Nov 01 12:57:30.040044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.039997 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" Nov 01 12:57:30.040044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040024 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" Nov 01 12:57:30.040044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040029 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" Nov 01 12:57:30.040044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040038 4484 cpu_manager.go:414] "containerMap: removing container" podUID="56fbf03b-f3e9-4994-9248-8f30e963748a" containerName="payload-pre-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040045 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fbf03b-f3e9-4994-9248-8f30e963748a" containerName="payload-pre-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040059 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040064 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040083 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerName="payload-pre-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040087 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerName="payload-pre-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040095 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040100 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040110 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerName="payload-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040115 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerName="payload-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040138 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c6720e72-5808-4a9e-bf48-2102e93d1620" containerName="revoke-keys" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040142 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6720e72-5808-4a9e-bf48-2102e93d1620" containerName="revoke-keys" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040150 4484 cpu_manager.go:414] "containerMap: removing container" podUID="237c99e9-2d6b-43e4-8858-8361d1fcc4f6" containerName="payload-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040155 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237c99e9-2d6b-43e4-8858-8361d1fcc4f6" containerName="payload-processing" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040162 4484 cpu_manager.go:414] "containerMap: removing container" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" Nov 01 12:57:30.040164 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040167 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040176 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040181 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040188 4484 cpu_manager.go:414] "containerMap: removing container" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040194 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040202 4484 cpu_manager.go:414] "containerMap: removing container" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040206 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040216 4484 cpu_manager.go:414] "containerMap: removing container" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040221 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040228 4484 cpu_manager.go:414] "containerMap: removing container" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040233 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040241 4484 cpu_manager.go:414] "containerMap: removing container" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040246 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040357 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3708e19-6de2-4ba2-a1c7-23e010ab81da" containerName="payload-pre-processing" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040372 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040385 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e820e6b-7262-4c30-acec-14251eaf26ed" containerName="payload-processing" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040395 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6720e72-5808-4a9e-bf48-2102e93d1620" containerName="revoke-keys" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040402 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="237c99e9-2d6b-43e4-8858-8361d1fcc4f6" containerName="payload-processing" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040407 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040413 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bea2e7e-a9c6-46b0-b930-8c769d293f66" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040420 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040425 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="106511c1-d902-40c3-a306-a5b8790400e4" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040432 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b14464-64a5-4ce3-8b8f-65504678c4c8" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040437 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040441 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="11835b7f-717d-48fa-8971-65b79dfa49d9" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040447 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a915fc-c906-43b1-8b47-cf0b54d97404" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040456 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" containerName="istio-proxy" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040462 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fbf03b-f3e9-4994-9248-8f30e963748a" containerName="payload-pre-processing" Nov 01 12:57:30.040651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.040469 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da1edfe-7017-482e-bb96-5aaf5d68bdd5" containerName="main" Nov 01 12:57:30.044442 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.044424 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.113018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.098114 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72078a\"" Nov 01 12:57:30.113018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.102113 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72078a-dockercfg-txktk\"" Nov 01 12:57:30.125323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.114966 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n"] Nov 01 12:57:30.212986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.202729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2ff\" (UniqueName: \"kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff\") pod \"payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.212986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.202842 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.256973 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:30.256932 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:57:30.303909 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.303835 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.304054 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.304006 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2ff\" (UniqueName: \"kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff\") pod \"payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.304613 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.304592 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.357614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.357585 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2ff\" (UniqueName: \"kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff\") pod \"payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.357729 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.357720 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:30.372371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.372350 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f659fa-cd16-4857-a348-3ed5d08c2714" path="/var/lib/kubelet/pods/76f659fa-cd16-4857-a348-3ed5d08c2714/volumes" Nov 01 12:57:30.372865 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.372850 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e410b77d-0691-47ef-a10d-b4cf73d4ce56" path="/var/lib/kubelet/pods/e410b77d-0691-47ef-a10d-b4cf73d4ce56/volumes" Nov 01 12:57:30.568504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.568435 4484 generic.go:356] "Generic (PLEG): container finished" podID="fe467197-9c0b-4c10-aa51-6504f92be21a" containerID="9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad" exitCode=0 Nov 01 12:57:30.568504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.568485 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" Nov 01 12:57:30.568678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.568539 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" event={"ID":"fe467197-9c0b-4c10-aa51-6504f92be21a","Type":"ContainerDied","Data":"9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad"} Nov 01 12:57:30.568678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.568596 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k" event={"ID":"fe467197-9c0b-4c10-aa51-6504f92be21a","Type":"ContainerDied","Data":"90f9768d5ffbbc9adfb98e38fc7fe1e9faf2bd77b10cc5aee2ff3c8cb2ad7edc"} Nov 01 12:57:30.568678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.568629 4484 scope.go:122] "RemoveContainer" containerID="9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad" Nov 01 12:57:30.578867 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.578849 4484 scope.go:122] "RemoveContainer" containerID="9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad" Nov 01 12:57:30.579187 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:30.579159 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad\": container with ID starting with 9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad not found: ID does not exist" containerID="9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad" Nov 01 12:57:30.579227 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.579200 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad"} err="failed to get container status \"9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad\": rpc error: code = NotFound desc = could not find container \"9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad\": container with ID starting with 9b092e4e32ef01724db109547cbc111d75d5b8f1a4cc186e4c732c7835c61dad not found: ID does not exist" Nov 01 12:57:30.735696 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.735668 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k"] Nov 01 12:57:30.741198 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.741173 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-8jr6k"] Nov 01 12:57:30.770549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:30.770521 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n"] Nov 01 12:57:31.438564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.438531 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb"] Nov 01 12:57:31.443077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.443057 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.494028 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.493997 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:57:31.494260 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.494243 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:57:31.498414 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.498387 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb"] Nov 01 12:57:31.524656 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.524626 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspts\" (UniqueName: \"kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.524799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.524774 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.524970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.524946 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.579928 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.579892 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" event={"ID":"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7","Type":"ContainerStarted","Data":"3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3"} Nov 01 12:57:31.580415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.579951 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" event={"ID":"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7","Type":"ContainerStarted","Data":"7ee7234d083ce48c7875c42e2a09b55cf1bc8f2def9cfce80a5359c0b6a27c1f"} Nov 01 12:57:31.580415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.580096 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:31.625958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.625936 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.626077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.626030 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.626178 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.626098 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspts\" (UniqueName: \"kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.626711 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.626686 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.628242 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.628224 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.679824 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.679801 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspts\" (UniqueName: \"kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts\") pod \"maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:31.688771 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.688692 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" podStartSLOduration=2.688679064 podStartE2EDuration="2.688679064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:57:31.687528965 +0000 UTC m=+2121.437471943" watchObservedRunningTime="2026-11-01 12:57:31.688679064 +0000 UTC m=+2121.438622042" Nov 01 12:57:31.756287 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:31.756264 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:32.159409 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:32.159371 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb"] Nov 01 12:57:32.165163 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:57:32.165121 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7724384e_c25d_403c_8ac1_3986cd143dbf.slice/crio-4e0575556783c6b6241e7a298f8a8fb8dff279721fd4a3142a0b707e10fd3add WatchSource:0}: Error finding container 4e0575556783c6b6241e7a298f8a8fb8dff279721fd4a3142a0b707e10fd3add: Status 404 returned error can't find the container with id 4e0575556783c6b6241e7a298f8a8fb8dff279721fd4a3142a0b707e10fd3add Nov 01 12:57:32.388324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:32.381466 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe467197-9c0b-4c10-aa51-6504f92be21a" path="/var/lib/kubelet/pods/fe467197-9c0b-4c10-aa51-6504f92be21a/volumes" Nov 01 12:57:32.432830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:32.432744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:32.587323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:32.584531 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:32.602717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:32.602680 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" event={"ID":"7724384e-c25d-403c-8ac1-3986cd143dbf","Type":"ContainerStarted","Data":"4e0575556783c6b6241e7a298f8a8fb8dff279721fd4a3142a0b707e10fd3add"} Nov 01 12:57:32.914674 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:32.914622 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:33.610397 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:33.610355 4484 generic.go:356] "Generic (PLEG): container finished" podID="7724384e-c25d-403c-8ac1-3986cd143dbf" containerID="70cef6b8c957859a7478afdfa7e9ace7e904e93e93aa005e78bcd6cfb6dfe291" exitCode=0 Nov 01 12:57:33.610779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:33.610441 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" event={"ID":"7724384e-c25d-403c-8ac1-3986cd143dbf","Type":"ContainerDied","Data":"70cef6b8c957859a7478afdfa7e9ace7e904e93e93aa005e78bcd6cfb6dfe291"} Nov 01 12:57:34.438553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.438507 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:34.811889 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.811863 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:34.976130 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.976102 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts\" (UniqueName: \"kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts\") pod \"7724384e-c25d-403c-8ac1-3986cd143dbf\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " Nov 01 12:57:34.976328 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.976236 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca\") pod \"7724384e-c25d-403c-8ac1-3986cd143dbf\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " Nov 01 12:57:34.976404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.976371 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca\") pod \"7724384e-c25d-403c-8ac1-3986cd143dbf\" (UID: \"7724384e-c25d-403c-8ac1-3986cd143dbf\") " Nov 01 12:57:34.976823 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.976793 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca" pod "7724384e-c25d-403c-8ac1-3986cd143dbf" (UID: "7724384e-c25d-403c-8ac1-3986cd143dbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:34.976961 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.976911 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:34.978625 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.978600 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca" pod "7724384e-c25d-403c-8ac1-3986cd143dbf" (UID: "7724384e-c25d-403c-8ac1-3986cd143dbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:57:34.978878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:34.978856 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts" pod "7724384e-c25d-403c-8ac1-3986cd143dbf" (UID: "7724384e-c25d-403c-8ac1-3986cd143dbf"). InnerVolumeSpecName "kube-api-access-xspts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:35.077666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:35.077603 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7724384e-c25d-403c-8ac1-3986cd143dbf-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:35.077666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:35.077630 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xspts\" (UniqueName: \"kubernetes.io/projected/7724384e-c25d-403c-8ac1-3986cd143dbf-kube-api-access-xspts\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:35.624668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:35.624637 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" Nov 01 12:57:35.624850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:35.624668 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb" event={"ID":"7724384e-c25d-403c-8ac1-3986cd143dbf","Type":"ContainerDied","Data":"4e0575556783c6b6241e7a298f8a8fb8dff279721fd4a3142a0b707e10fd3add"} Nov 01 12:57:35.624850 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:35.624715 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e0575556783c6b6241e7a298f8a8fb8dff279721fd4a3142a0b707e10fd3add" Nov 01 12:57:36.517081 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:36.517047 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:36.780140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:36.780065 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:36.924880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:36.924847 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:37.107983 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:37.107902 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd"] Nov 01 12:57:37.150834 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:37.150799 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f893c2-xmlsd"] Nov 01 12:57:37.501794 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:37.501765 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:38.371053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:38.371015 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63ad19f-2eae-4ff1-8b1d-2a6173622cdd" path="/var/lib/kubelet/pods/c63ad19f-2eae-4ff1-8b1d-2a6173622cdd/volumes" Nov 01 12:57:38.826143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:38.826112 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:38.979361 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:38.979325 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:42.220332 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:42.220280 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:42.359952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:42.359910 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:42.606398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:42.606372 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 12:57:42.973870 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:42.973772 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:42.985600 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:42.985573 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj"] Nov 01 12:57:43.115738 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:43.115707 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f"] Nov 01 12:57:43.364847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:43.364801 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:43.507288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:43.507259 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:44.974202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:44.974139 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:45.119510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:45.119476 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:45.337723 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:45.337697 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:45.347156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:45.347119 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8"] Nov 01 12:57:45.347430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:45.347407 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" containerID="cri-o://d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685" gracePeriod=30 Nov 01 12:57:47.985915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:47.985866 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" podUID="b9fa3971-744a-4ad4-bd5d-631db925c944" containerName="payload-processing" containerID="cri-o://dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb" gracePeriod=25 Nov 01 12:57:48.116203 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.116167 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" podUID="acdce95a-17d2-40f4-a33d-e4f1af50acef" containerName="payload-pre-processing" containerID="cri-o://c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673" gracePeriod=25 Nov 01 12:57:48.147248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.147219 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2"] Nov 01 12:57:48.147876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.147858 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7724384e-c25d-403c-8ac1-3986cd143dbf" containerName="revoke-keys" Nov 01 12:57:48.147876 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.147875 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7724384e-c25d-403c-8ac1-3986cd143dbf" containerName="revoke-keys" Nov 01 12:57:48.148010 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.147956 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7724384e-c25d-403c-8ac1-3986cd143dbf" containerName="revoke-keys" Nov 01 12:57:48.151701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.151679 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.201132 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.201086 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:57:48.246544 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.246513 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:57:48.249973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.249951 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2"] Nov 01 12:57:48.272615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.272590 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:57:48.278430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.278398 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57d9\" (UniqueName: \"kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.278574 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.278483 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.278635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.278578 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.379837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.379789 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2\" (UniqueName: \"kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2\") pod \"b9fa3971-744a-4ad4-bd5d-631db925c944\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " Nov 01 12:57:48.381228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.379951 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume\") pod \"b9fa3971-744a-4ad4-bd5d-631db925c944\" (UID: \"b9fa3971-744a-4ad4-bd5d-631db925c944\") " Nov 01 12:57:48.381228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.380238 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.381228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.380552 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57d9\" (UniqueName: \"kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.381228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.380605 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume" pod "b9fa3971-744a-4ad4-bd5d-631db925c944" (UID: "b9fa3971-744a-4ad4-bd5d-631db925c944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:48.381228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.380815 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.381228 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.380943 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fa3971-744a-4ad4-bd5d-631db925c944-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:48.381718 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.381697 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.382971 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.382942 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2" pod "b9fa3971-744a-4ad4-bd5d-631db925c944" (UID: "b9fa3971-744a-4ad4-bd5d-631db925c944"). InnerVolumeSpecName "kube-api-access-ff5w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:48.384731 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.384701 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.406354 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.406334 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:57:48.439981 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.439954 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57d9\" (UniqueName: \"kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9\") pod \"maas-api-revoke-sub-34b56a7d-nxdj2\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.481763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.481737 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk\" (UniqueName: \"kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk\") pod \"acdce95a-17d2-40f4-a33d-e4f1af50acef\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " Nov 01 12:57:48.481917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.481903 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume\") pod \"acdce95a-17d2-40f4-a33d-e4f1af50acef\" (UID: \"acdce95a-17d2-40f4-a33d-e4f1af50acef\") " Nov 01 12:57:48.482259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.482235 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume" pod "acdce95a-17d2-40f4-a33d-e4f1af50acef" (UID: "acdce95a-17d2-40f4-a33d-e4f1af50acef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:48.482443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.482426 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5w2\" (UniqueName: \"kubernetes.io/projected/b9fa3971-744a-4ad4-bd5d-631db925c944-kube-api-access-ff5w2\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:48.482513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.482448 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdce95a-17d2-40f4-a33d-e4f1af50acef-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:48.483618 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.483600 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk" pod "acdce95a-17d2-40f4-a33d-e4f1af50acef" (UID: "acdce95a-17d2-40f4-a33d-e4f1af50acef"). InnerVolumeSpecName "kube-api-access-nqjmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:48.513264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.513212 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:48.584334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.584283 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjmk\" (UniqueName: \"kubernetes.io/projected/acdce95a-17d2-40f4-a33d-e4f1af50acef-kube-api-access-nqjmk\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:48.719671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.719633 4484 generic.go:356] "Generic (PLEG): container finished" podID="b9fa3971-744a-4ad4-bd5d-631db925c944" containerID="dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb" exitCode=0 Nov 01 12:57:48.719842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.719685 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" Nov 01 12:57:48.719842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.719715 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" event={"ID":"b9fa3971-744a-4ad4-bd5d-631db925c944","Type":"ContainerDied","Data":"dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb"} Nov 01 12:57:48.719842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.719774 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj" event={"ID":"b9fa3971-744a-4ad4-bd5d-631db925c944","Type":"ContainerDied","Data":"72df339bdbaa465f12f61f13d31967484fc7dff257a312df64b048646df6172c"} Nov 01 12:57:48.719842 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.719798 4484 scope.go:122] "RemoveContainer" containerID="dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb" Nov 01 12:57:48.721465 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.721440 4484 generic.go:356] "Generic (PLEG): container finished" podID="acdce95a-17d2-40f4-a33d-e4f1af50acef" containerID="c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673" exitCode=0 Nov 01 12:57:48.721591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.721474 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" Nov 01 12:57:48.721591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.721532 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" event={"ID":"acdce95a-17d2-40f4-a33d-e4f1af50acef","Type":"ContainerDied","Data":"c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673"} Nov 01 12:57:48.721591 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.721567 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f" event={"ID":"acdce95a-17d2-40f4-a33d-e4f1af50acef","Type":"ContainerDied","Data":"73269140571b308d28e27b3cf3831b450014c2eaf355f22f63589296c4a2743f"} Nov 01 12:57:48.731413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.731334 4484 scope.go:122] "RemoveContainer" containerID="dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb" Nov 01 12:57:48.731772 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:48.731751 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb\": container with ID starting with dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb not found: ID does not exist" containerID="dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb" Nov 01 12:57:48.731880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.731783 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb"} err="failed to get container status \"dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb\": rpc error: code = NotFound desc = could not find container \"dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb\": container with ID starting with dfbf66d040b64ee59ca2727b1ca5f425c2e29b04145fca9ce099e255d22decbb not found: ID does not exist" Nov 01 12:57:48.731880 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.731806 4484 scope.go:122] "RemoveContainer" containerID="c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673" Nov 01 12:57:48.741427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.741409 4484 scope.go:122] "RemoveContainer" containerID="c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673" Nov 01 12:57:48.741708 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:48.741691 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673\": container with ID starting with c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673 not found: ID does not exist" containerID="c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673" Nov 01 12:57:48.741754 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.741713 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673"} err="failed to get container status \"c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673\": rpc error: code = NotFound desc = could not find container \"c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673\": container with ID starting with c42f3593d93f0db63969e2a96b19dce50c50611a29e5d2c9f4cc46ee44b3c673 not found: ID does not exist" Nov 01 12:57:48.873623 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.873549 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj"] Nov 01 12:57:48.919369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.919340 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0a0610-5f94c88dd5-n7fmj"] Nov 01 12:57:48.922418 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.922397 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2"] Nov 01 12:57:48.937370 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:57:48.937342 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4967d4d_cda5_4e04_a2e6_a69b6da666fc.slice/crio-2ae61185e5b956d999b12428d4ee13d34a1bcaac050664f76aaec3878dec8807 WatchSource:0}: Error finding container 2ae61185e5b956d999b12428d4ee13d34a1bcaac050664f76aaec3878dec8807: Status 404 returned error can't find the container with id 2ae61185e5b956d999b12428d4ee13d34a1bcaac050664f76aaec3878dec8807 Nov 01 12:57:48.976916 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.976894 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f"] Nov 01 12:57:48.985417 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:48.985395 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0a0610-7577bcdd7-rkj4f"] Nov 01 12:57:49.215022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.214953 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:49.316014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.315978 4484 patch_prober.go:29] interesting pod/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 12:57:49.316206 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.316037 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 12:57:49.569200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.569172 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:57:49.738698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.738651 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" event={"ID":"c4967d4d-cda5-4e04-a2e6-a69b6da666fc","Type":"ContainerStarted","Data":"f37803a80b7b29810d4b8751c3f4e4d289c4179450e9f1e5903c6c8a7ecc7518"} Nov 01 12:57:49.738889 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.738751 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" event={"ID":"c4967d4d-cda5-4e04-a2e6-a69b6da666fc","Type":"ContainerStarted","Data":"2ae61185e5b956d999b12428d4ee13d34a1bcaac050664f76aaec3878dec8807"} Nov 01 12:57:49.849413 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:49.849326 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" podStartSLOduration=1.849296314 podStartE2EDuration="1.849296314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:57:49.847239675 +0000 UTC m=+2139.597182653" watchObservedRunningTime="2026-11-01 12:57:49.849296314 +0000 UTC m=+2139.599239292" Nov 01 12:57:50.369958 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.369930 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdce95a-17d2-40f4-a33d-e4f1af50acef" path="/var/lib/kubelet/pods/acdce95a-17d2-40f4-a33d-e4f1af50acef/volumes" Nov 01 12:57:50.370325 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.370295 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fa3971-744a-4ad4-bd5d-631db925c944" path="/var/lib/kubelet/pods/b9fa3971-744a-4ad4-bd5d-631db925c944/volumes" Nov 01 12:57:50.660427 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.660398 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:57:50.749288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.749256 4484 generic.go:356] "Generic (PLEG): container finished" podID="baae4295-9113-41d7-a6db-1d030f4590e5" containerID="d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685" exitCode=0 Nov 01 12:57:50.749482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.749326 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" Nov 01 12:57:50.749482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.749347 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" event={"ID":"baae4295-9113-41d7-a6db-1d030f4590e5","Type":"ContainerDied","Data":"d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685"} Nov 01 12:57:50.749482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.749405 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8" event={"ID":"baae4295-9113-41d7-a6db-1d030f4590e5","Type":"ContainerDied","Data":"4b993e904f565b9f8ff57d2842cf9c155ed5dbce8f05c04fbd15d6fc17e17c59"} Nov 01 12:57:50.749482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.749428 4484 scope.go:122] "RemoveContainer" containerID="d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685" Nov 01 12:57:50.751193 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.751171 4484 generic.go:356] "Generic (PLEG): container finished" podID="c4967d4d-cda5-4e04-a2e6-a69b6da666fc" containerID="f37803a80b7b29810d4b8751c3f4e4d289c4179450e9f1e5903c6c8a7ecc7518" exitCode=0 Nov 01 12:57:50.751324 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.751242 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" event={"ID":"c4967d4d-cda5-4e04-a2e6-a69b6da666fc","Type":"ContainerDied","Data":"f37803a80b7b29810d4b8751c3f4e4d289c4179450e9f1e5903c6c8a7ecc7518"} Nov 01 12:57:50.759767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.759752 4484 scope.go:122] "RemoveContainer" containerID="78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82" Nov 01 12:57:50.768677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.768660 4484 scope.go:122] "RemoveContainer" containerID="d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685" Nov 01 12:57:50.768937 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:50.768917 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685\": container with ID starting with d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685 not found: ID does not exist" containerID="d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685" Nov 01 12:57:50.768986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.768945 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685"} err="failed to get container status \"d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685\": rpc error: code = NotFound desc = could not find container \"d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685\": container with ID starting with d69da192132f932053d742fd40ec37fde851cbcdf2e1ebf1924f0bfdbbda1685 not found: ID does not exist" Nov 01 12:57:50.768986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.768961 4484 scope.go:122] "RemoveContainer" containerID="78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82" Nov 01 12:57:50.769184 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:57:50.769165 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82\": container with ID starting with 78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82 not found: ID does not exist" containerID="78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82" Nov 01 12:57:50.769234 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.769189 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82"} err="failed to get container status \"78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82\": rpc error: code = NotFound desc = could not find container \"78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82\": container with ID starting with 78f1f044682e59b45b5a975b423e97a368c93f8c8138d06d6184022bb520df82 not found: ID does not exist" Nov 01 12:57:50.812633 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812606 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.812725 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812644 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.812789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812737 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.812789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812785 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.812899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812822 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.812899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812825 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:50.813014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812934 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.813014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812976 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.813014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.812999 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.813177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813079 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:50.813177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813132 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert\") pod \"baae4295-9113-41d7-a6db-1d030f4590e5\" (UID: \"baae4295-9113-41d7-a6db-1d030f4590e5\") " Nov 01 12:57:50.813293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813196 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:50.813293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813252 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:50.813527 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813503 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:50.813716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813694 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.813789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813722 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.813789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813737 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.813789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813752 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4295-9113-41d7-a6db-1d030f4590e5-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.813789 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.813766 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.814927 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.814902 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "kube-api-access-6j4tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:50.815037 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.814930 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:57:50.815037 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.814988 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:57:50.815183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.815161 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token" pod "baae4295-9113-41d7-a6db-1d030f4590e5" (UID: "baae4295-9113-41d7-a6db-1d030f4590e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:50.914686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.914615 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4295-9113-41d7-a6db-1d030f4590e5-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.914686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.914641 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.914686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.914654 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4295-9113-41d7-a6db-1d030f4590e5-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:50.914686 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:50.914667 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j4tq\" (UniqueName: \"kubernetes.io/projected/baae4295-9113-41d7-a6db-1d030f4590e5-kube-api-access-6j4tq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:51.164202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:51.164164 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8"] Nov 01 12:57:51.168405 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:51.168344 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f893c2-openshift-default-78c956c74c-lcfk8"] Nov 01 12:57:51.904687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:51.904662 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:52.032053 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.031974 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca\") pod \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " Nov 01 12:57:52.032219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.032072 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9\" (UniqueName: \"kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9\") pod \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " Nov 01 12:57:52.032219 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.032132 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca\") pod \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\" (UID: \"c4967d4d-cda5-4e04-a2e6-a69b6da666fc\") " Nov 01 12:57:52.032392 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.032367 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca" pod "c4967d4d-cda5-4e04-a2e6-a69b6da666fc" (UID: "c4967d4d-cda5-4e04-a2e6-a69b6da666fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:57:52.032888 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.032859 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:52.034752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.034728 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca" pod "c4967d4d-cda5-4e04-a2e6-a69b6da666fc" (UID: "c4967d4d-cda5-4e04-a2e6-a69b6da666fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:57:52.034833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.034755 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9" pod "c4967d4d-cda5-4e04-a2e6-a69b6da666fc" (UID: "c4967d4d-cda5-4e04-a2e6-a69b6da666fc"). InnerVolumeSpecName "kube-api-access-s57d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:57:52.134605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.134575 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57d9\" (UniqueName: \"kubernetes.io/projected/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-kube-api-access-s57d9\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:52.134605 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.134602 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4967d4d-cda5-4e04-a2e6-a69b6da666fc-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:57:52.370363 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.370276 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" path="/var/lib/kubelet/pods/baae4295-9113-41d7-a6db-1d030f4590e5/volumes" Nov 01 12:57:52.769100 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.769071 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" Nov 01 12:57:52.769271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.769078 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2" event={"ID":"c4967d4d-cda5-4e04-a2e6-a69b6da666fc","Type":"ContainerDied","Data":"2ae61185e5b956d999b12428d4ee13d34a1bcaac050664f76aaec3878dec8807"} Nov 01 12:57:52.769271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:52.769190 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae61185e5b956d999b12428d4ee13d34a1bcaac050664f76aaec3878dec8807" Nov 01 12:57:55.322370 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:55.322050 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:59.214094 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:59.214058 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85"] Nov 01 12:57:59.377652 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:59.377616 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc"] Nov 01 12:57:59.591767 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:59.591733 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:57:59.724450 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:57:59.724419 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:00.251273 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:00.251237 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:58:01.368435 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:01.368398 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:02.222346 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:02.222287 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:02.345917 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:02.345881 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:04.215320 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.215255 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" podUID="7140dee5-84f0-4805-962f-f22bffa8f545" containerName="payload-processing" containerID="cri-o://1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b" gracePeriod=25 Nov 01 12:58:04.378316 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.378263 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" podUID="80896d05-d706-4abb-a93a-7cd061b4e36e" containerName="payload-pre-processing" containerID="cri-o://4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9" gracePeriod=25 Nov 01 12:58:04.509664 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.509641 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:58:04.606530 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.606498 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume\") pod \"7140dee5-84f0-4805-962f-f22bffa8f545\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " Nov 01 12:58:04.606687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.606592 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7\" (UniqueName: \"kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7\") pod \"7140dee5-84f0-4805-962f-f22bffa8f545\" (UID: \"7140dee5-84f0-4805-962f-f22bffa8f545\") " Nov 01 12:58:04.606990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.606955 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume" pod "7140dee5-84f0-4805-962f-f22bffa8f545" (UID: "7140dee5-84f0-4805-962f-f22bffa8f545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:58:04.607176 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.607155 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7140dee5-84f0-4805-962f-f22bffa8f545-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:04.608737 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.608710 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7" pod "7140dee5-84f0-4805-962f-f22bffa8f545" (UID: "7140dee5-84f0-4805-962f-f22bffa8f545"). InnerVolumeSpecName "kube-api-access-wvnn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:58:04.626839 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.626810 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:58:04.708429 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.708398 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume\") pod \"80896d05-d706-4abb-a93a-7cd061b4e36e\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " Nov 01 12:58:04.708575 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.708550 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst\" (UniqueName: \"kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst\") pod \"80896d05-d706-4abb-a93a-7cd061b4e36e\" (UID: \"80896d05-d706-4abb-a93a-7cd061b4e36e\") " Nov 01 12:58:04.708754 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.708732 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume" pod "80896d05-d706-4abb-a93a-7cd061b4e36e" (UID: "80896d05-d706-4abb-a93a-7cd061b4e36e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:58:04.709016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.708999 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvnn7\" (UniqueName: \"kubernetes.io/projected/7140dee5-84f0-4805-962f-f22bffa8f545-kube-api-access-wvnn7\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:04.709062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.709019 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80896d05-d706-4abb-a93a-7cd061b4e36e-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:04.710667 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.710649 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst" pod "80896d05-d706-4abb-a93a-7cd061b4e36e" (UID: "80896d05-d706-4abb-a93a-7cd061b4e36e"). InnerVolumeSpecName "kube-api-access-xvtst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:58:04.810167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.810096 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvtst\" (UniqueName: \"kubernetes.io/projected/80896d05-d706-4abb-a93a-7cd061b4e36e-kube-api-access-xvtst\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:04.864638 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.864609 4484 generic.go:356] "Generic (PLEG): container finished" podID="7140dee5-84f0-4805-962f-f22bffa8f545" containerID="1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b" exitCode=0 Nov 01 12:58:04.864802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.864667 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" Nov 01 12:58:04.864802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.864693 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" event={"ID":"7140dee5-84f0-4805-962f-f22bffa8f545","Type":"ContainerDied","Data":"1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b"} Nov 01 12:58:04.864802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.864755 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85" event={"ID":"7140dee5-84f0-4805-962f-f22bffa8f545","Type":"ContainerDied","Data":"fba5ccea41fd8b3cc94c14b89e8e2cc3650372c181d9f572f0b20ddf2fc05e7c"} Nov 01 12:58:04.864802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.864779 4484 scope.go:122] "RemoveContainer" containerID="1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b" Nov 01 12:58:04.866386 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.866362 4484 generic.go:356] "Generic (PLEG): container finished" podID="80896d05-d706-4abb-a93a-7cd061b4e36e" containerID="4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9" exitCode=0 Nov 01 12:58:04.866557 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.866392 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" event={"ID":"80896d05-d706-4abb-a93a-7cd061b4e36e","Type":"ContainerDied","Data":"4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9"} Nov 01 12:58:04.866651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.866597 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" event={"ID":"80896d05-d706-4abb-a93a-7cd061b4e36e","Type":"ContainerDied","Data":"0eacba522b2a764f22b87f384a65e875a894216405f0571cc5d96d4d24dbc6bf"} Nov 01 12:58:04.866651 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.866407 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc" Nov 01 12:58:04.877430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.877402 4484 scope.go:122] "RemoveContainer" containerID="1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b" Nov 01 12:58:04.877930 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:04.877903 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b\": container with ID starting with 1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b not found: ID does not exist" containerID="1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b" Nov 01 12:58:04.878038 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.877938 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b"} err="failed to get container status \"1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b\": rpc error: code = NotFound desc = could not find container \"1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b\": container with ID starting with 1fbf1bd51ee9d6958f3072a1caa31436596f7bdc29ce7f50cbdc126ecf0c8e1b not found: ID does not exist" Nov 01 12:58:04.878038 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.877961 4484 scope.go:122] "RemoveContainer" containerID="4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9" Nov 01 12:58:04.886631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.886614 4484 scope.go:122] "RemoveContainer" containerID="4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9" Nov 01 12:58:04.886904 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:04.886886 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9\": container with ID starting with 4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9 not found: ID does not exist" containerID="4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9" Nov 01 12:58:04.886952 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:04.886912 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9"} err="failed to get container status \"4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9\": rpc error: code = NotFound desc = could not find container \"4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9\": container with ID starting with 4a85fe2d581c188118c878f0e94995f309533db2bbd43f296d2e0e8eb78f68a9 not found: ID does not exist" Nov 01 12:58:05.017488 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:05.017461 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc"] Nov 01 12:58:05.020531 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:05.020504 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6c650b-766bd9fc87-kbklc"] Nov 01 12:58:05.123940 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:05.123878 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85"] Nov 01 12:58:05.133247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:05.133207 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6c650b-7df8fb549f-8vk85"] Nov 01 12:58:06.370125 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:06.370089 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7140dee5-84f0-4805-962f-f22bffa8f545" path="/var/lib/kubelet/pods/7140dee5-84f0-4805-962f-f22bffa8f545/volumes" Nov 01 12:58:06.370592 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:06.370576 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80896d05-d706-4abb-a93a-7cd061b4e36e" path="/var/lib/kubelet/pods/80896d05-d706-4abb-a93a-7cd061b4e36e/volumes" Nov 01 12:58:07.262580 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:07.262539 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:12.189443 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:12.189414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:12.391748 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:12.391671 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:13.225225 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:13.225191 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:15.053216 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:15.053176 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:15.178008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:15.177977 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:18.960559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:18.960526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:23.508945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.508910 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9"] Nov 01 12:58:23.509394 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.509104 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" podUID="1eb9367c-4699-4cba-af72-48ed1912e2be" containerName="maas-api" containerID="cri-o://0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93" gracePeriod=30 Nov 01 12:58:23.597619 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.597591 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:23.818230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.818193 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:58:23.854430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.854399 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca\") pod \"1eb9367c-4699-4cba-af72-48ed1912e2be\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " Nov 01 12:58:23.854564 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.854551 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl\" (UniqueName: \"kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl\") pod \"1eb9367c-4699-4cba-af72-48ed1912e2be\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " Nov 01 12:58:23.854608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.854593 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs\") pod \"1eb9367c-4699-4cba-af72-48ed1912e2be\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " Nov 01 12:58:23.854649 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.854636 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\") pod \"1eb9367c-4699-4cba-af72-48ed1912e2be\" (UID: \"1eb9367c-4699-4cba-af72-48ed1912e2be\") " Nov 01 12:58:23.854715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.854693 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca" pod "1eb9367c-4699-4cba-af72-48ed1912e2be" (UID: "1eb9367c-4699-4cba-af72-48ed1912e2be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:58:23.855101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.855081 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb9367c-4699-4cba-af72-48ed1912e2be-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:23.856702 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.856672 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs" pod "1eb9367c-4699-4cba-af72-48ed1912e2be" (UID: "1eb9367c-4699-4cba-af72-48ed1912e2be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:58:23.856799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.856730 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls" pod "1eb9367c-4699-4cba-af72-48ed1912e2be" (UID: "1eb9367c-4699-4cba-af72-48ed1912e2be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:58:23.856799 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.856753 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl" pod "1eb9367c-4699-4cba-af72-48ed1912e2be" (UID: "1eb9367c-4699-4cba-af72-48ed1912e2be"). InnerVolumeSpecName "kube-api-access-kqzfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:58:23.956264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.956224 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqzfl\" (UniqueName: \"kubernetes.io/projected/1eb9367c-4699-4cba-af72-48ed1912e2be-kube-api-access-kqzfl\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:23.956264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.956268 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:23.956547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.956285 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eb9367c-4699-4cba-af72-48ed1912e2be-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:23.991272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.991234 4484 generic.go:356] "Generic (PLEG): container finished" podID="1eb9367c-4699-4cba-af72-48ed1912e2be" containerID="0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93" exitCode=0 Nov 01 12:58:23.991462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.991287 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" Nov 01 12:58:23.991462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.991318 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" event={"ID":"1eb9367c-4699-4cba-af72-48ed1912e2be","Type":"ContainerDied","Data":"0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93"} Nov 01 12:58:23.991462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.991380 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9" event={"ID":"1eb9367c-4699-4cba-af72-48ed1912e2be","Type":"ContainerDied","Data":"7310a421b5cbb1236c3f0167cc629dfcb08c1447fdd5168aa25b76ea914e71fb"} Nov 01 12:58:23.991462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:23.991413 4484 scope.go:122] "RemoveContainer" containerID="0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93" Nov 01 12:58:24.004177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:24.004159 4484 scope.go:122] "RemoveContainer" containerID="0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93" Nov 01 12:58:24.004515 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:24.004492 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93\": container with ID starting with 0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93 not found: ID does not exist" containerID="0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93" Nov 01 12:58:24.004604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:24.004524 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93"} err="failed to get container status \"0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93\": rpc error: code = NotFound desc = could not find container \"0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93\": container with ID starting with 0a761fa6bdc77ec4fa35ec234463b3b9790b24f3dbeafc6d9d57f202d1e66a93 not found: ID does not exist" Nov 01 12:58:24.144292 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:24.144211 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9"] Nov 01 12:58:24.150034 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:24.150008 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6c650b-5db5c4998f-l6rk9"] Nov 01 12:58:24.369263 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:24.369229 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb9367c-4699-4cba-af72-48ed1912e2be" path="/var/lib/kubelet/pods/1eb9367c-4699-4cba-af72-48ed1912e2be/volumes" Nov 01 12:58:25.312167 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:25.312135 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:25.654016 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:25.653935 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:25.774790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:25.774755 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:26.287615 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:26.287583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:30.251831 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:30.251791 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:58:31.462549 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:31.462518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:31.591809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:31.591778 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:32.078022 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:32.077988 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:58:32.571358 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:32.571325 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:32.709703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:32.709670 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:32.926405 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:32.926316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:33.842119 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842087 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk"] Nov 01 12:58:33.842635 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842621 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7140dee5-84f0-4805-962f-f22bffa8f545" containerName="payload-processing" Nov 01 12:58:33.842678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842636 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7140dee5-84f0-4805-962f-f22bffa8f545" containerName="payload-processing" Nov 01 12:58:33.842678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842658 4484 cpu_manager.go:414] "containerMap: removing container" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" Nov 01 12:58:33.842678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842663 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" Nov 01 12:58:33.842678 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842675 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1eb9367c-4699-4cba-af72-48ed1912e2be" containerName="maas-api" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842680 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb9367c-4699-4cba-af72-48ed1912e2be" containerName="maas-api" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842695 4484 cpu_manager.go:414] "containerMap: removing container" podUID="acdce95a-17d2-40f4-a33d-e4f1af50acef" containerName="payload-pre-processing" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842700 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdce95a-17d2-40f4-a33d-e4f1af50acef" containerName="payload-pre-processing" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842709 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b9fa3971-744a-4ad4-bd5d-631db925c944" containerName="payload-processing" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842713 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fa3971-744a-4ad4-bd5d-631db925c944" containerName="payload-processing" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842722 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c4967d4d-cda5-4e04-a2e6-a69b6da666fc" containerName="revoke-keys" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842727 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4967d4d-cda5-4e04-a2e6-a69b6da666fc" containerName="revoke-keys" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842737 4484 cpu_manager.go:414] "containerMap: removing container" podUID="80896d05-d706-4abb-a93a-7cd061b4e36e" containerName="payload-pre-processing" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842741 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80896d05-d706-4abb-a93a-7cd061b4e36e" containerName="payload-pre-processing" Nov 01 12:58:33.842809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842808 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842820 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb9367c-4699-4cba-af72-48ed1912e2be" containerName="maas-api" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842826 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842834 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdce95a-17d2-40f4-a33d-e4f1af50acef" containerName="payload-pre-processing" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842841 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fa3971-744a-4ad4-bd5d-631db925c944" containerName="payload-processing" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842848 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4967d4d-cda5-4e04-a2e6-a69b6da666fc" containerName="revoke-keys" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842853 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="80896d05-d706-4abb-a93a-7cd061b4e36e" containerName="payload-pre-processing" Nov 01 12:58:33.843107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.842860 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7140dee5-84f0-4805-962f-f22bffa8f545" containerName="payload-processing" Nov 01 12:58:33.847989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.847971 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:33.897050 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.897030 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:58:33.897159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.897120 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:58:33.899695 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.899672 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk"] Nov 01 12:58:33.989802 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.989776 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:33.989953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.989837 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dk4\" (UniqueName: \"kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:33.989953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:33.989935 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.091113 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.091079 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dk4\" (UniqueName: \"kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.091271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.091134 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.091271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.091239 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.091841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.091815 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.093612 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.093560 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.145160 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.145131 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dk4\" (UniqueName: \"kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.164262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.164239 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:34.534899 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:34.534860 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk"] Nov 01 12:58:34.541822 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:58:34.541795 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7564a4b_f381_46d0_9f27_8ad59e0a1050.slice/crio-9087f560394ed138a39a9e94d38c943ccc20ad4457ccb3bc6d06e094547fbdb4 WatchSource:0}: Error finding container 9087f560394ed138a39a9e94d38c943ccc20ad4457ccb3bc6d06e094547fbdb4: Status 404 returned error can't find the container with id 9087f560394ed138a39a9e94d38c943ccc20ad4457ccb3bc6d06e094547fbdb4 Nov 01 12:58:35.081756 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:35.081726 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" event={"ID":"e7564a4b-f381-46d0-9f27-8ad59e0a1050","Type":"ContainerStarted","Data":"cca226972b54678c1926dccaadee001017bb9dac6358972783ec082f3985704c"} Nov 01 12:58:35.082121 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:35.081781 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" event={"ID":"e7564a4b-f381-46d0-9f27-8ad59e0a1050","Type":"ContainerStarted","Data":"9087f560394ed138a39a9e94d38c943ccc20ad4457ccb3bc6d06e094547fbdb4"} Nov 01 12:58:35.189990 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:35.189908 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" podStartSLOduration=2.189894404 podStartE2EDuration="2.189894404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:58:35.188553604 +0000 UTC m=+2184.938496582" watchObservedRunningTime="2026-11-01 12:58:35.189894404 +0000 UTC m=+2184.939837381" Nov 01 12:58:36.089587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:36.089553 4484 generic.go:356] "Generic (PLEG): container finished" podID="e7564a4b-f381-46d0-9f27-8ad59e0a1050" containerID="cca226972b54678c1926dccaadee001017bb9dac6358972783ec082f3985704c" exitCode=0 Nov 01 12:58:36.089993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:36.089650 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" event={"ID":"e7564a4b-f381-46d0-9f27-8ad59e0a1050","Type":"ContainerDied","Data":"cca226972b54678c1926dccaadee001017bb9dac6358972783ec082f3985704c"} Nov 01 12:58:37.245684 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.245658 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:37.290947 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.290922 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:37.328079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.328054 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca\") pod \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " Nov 01 12:58:37.328182 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.328099 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca\") pod \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " Nov 01 12:58:37.328223 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.328197 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4\" (UniqueName: \"kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4\") pod \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\" (UID: \"e7564a4b-f381-46d0-9f27-8ad59e0a1050\") " Nov 01 12:58:37.328499 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.328465 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca" pod "e7564a4b-f381-46d0-9f27-8ad59e0a1050" (UID: "e7564a4b-f381-46d0-9f27-8ad59e0a1050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:58:37.328639 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.328624 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:37.330172 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.330152 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4" pod "e7564a4b-f381-46d0-9f27-8ad59e0a1050" (UID: "e7564a4b-f381-46d0-9f27-8ad59e0a1050"). InnerVolumeSpecName "kube-api-access-r7dk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:58:37.330252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.330216 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca" pod "e7564a4b-f381-46d0-9f27-8ad59e0a1050" (UID: "e7564a4b-f381-46d0-9f27-8ad59e0a1050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:58:37.429970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.429892 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dk4\" (UniqueName: \"kubernetes.io/projected/e7564a4b-f381-46d0-9f27-8ad59e0a1050-kube-api-access-r7dk4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:37.429970 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:37.429922 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7564a4b-f381-46d0-9f27-8ad59e0a1050-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:38.103837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:38.103806 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" Nov 01 12:58:38.104019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:38.103847 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk" event={"ID":"e7564a4b-f381-46d0-9f27-8ad59e0a1050","Type":"ContainerDied","Data":"9087f560394ed138a39a9e94d38c943ccc20ad4457ccb3bc6d06e094547fbdb4"} Nov 01 12:58:38.104019 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:38.103894 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9087f560394ed138a39a9e94d38c943ccc20ad4457ccb3bc6d06e094547fbdb4" Nov 01 12:58:43.006953 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:43.006897 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:45.507348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:45.507296 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv"] Nov 01 12:58:45.746129 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:45.746090 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:45.867286 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:45.867204 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:48.525902 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:48.525870 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:48.778515 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:48.778426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:48.905357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:48.905290 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:50.509116 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:50.509064 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" podUID="b85d3fb7-bd4a-41f8-8274-3efda26c7f03" containerName="payload-pre-processing" containerID="cri-o://7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea" gracePeriod=25 Nov 01 12:58:50.795787 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:50.795759 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:58:50.908183 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:50.908152 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh\" (UniqueName: \"kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh\") pod \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " Nov 01 12:58:50.908396 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:50.908218 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume\") pod \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\" (UID: \"b85d3fb7-bd4a-41f8-8274-3efda26c7f03\") " Nov 01 12:58:50.908887 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:50.908862 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume" pod "b85d3fb7-bd4a-41f8-8274-3efda26c7f03" (UID: "b85d3fb7-bd4a-41f8-8274-3efda26c7f03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:58:50.910881 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:50.910856 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh" pod "b85d3fb7-bd4a-41f8-8274-3efda26c7f03" (UID: "b85d3fb7-bd4a-41f8-8274-3efda26c7f03"). InnerVolumeSpecName "kube-api-access-2dnrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:58:51.009717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.009685 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnrh\" (UniqueName: \"kubernetes.io/projected/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-kube-api-access-2dnrh\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:51.009717 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.009714 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b85d3fb7-bd4a-41f8-8274-3efda26c7f03-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:51.185988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.185897 4484 generic.go:356] "Generic (PLEG): container finished" podID="b85d3fb7-bd4a-41f8-8274-3efda26c7f03" containerID="7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea" exitCode=0 Nov 01 12:58:51.185988 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.185947 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" Nov 01 12:58:51.186248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.185986 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" event={"ID":"b85d3fb7-bd4a-41f8-8274-3efda26c7f03","Type":"ContainerDied","Data":"7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea"} Nov 01 12:58:51.186248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.186045 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv" event={"ID":"b85d3fb7-bd4a-41f8-8274-3efda26c7f03","Type":"ContainerDied","Data":"8fd45624d9780a4e9f21391a7ff390d8d53c1f6815fc4a3dbae9a056f6c98c95"} Nov 01 12:58:51.186248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.186069 4484 scope.go:122] "RemoveContainer" containerID="7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea" Nov 01 12:58:51.199148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.199128 4484 scope.go:122] "RemoveContainer" containerID="7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea" Nov 01 12:58:51.199675 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:51.199650 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea\": container with ID starting with 7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea not found: ID does not exist" containerID="7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea" Nov 01 12:58:51.199779 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.199681 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea"} err="failed to get container status \"7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea\": rpc error: code = NotFound desc = could not find container \"7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea\": container with ID starting with 7656b9b9f7f8a64d11a86934e32dd5271fc151b3270ce5bf88bbf62b724647ea not found: ID does not exist" Nov 01 12:58:51.338562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.338532 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv"] Nov 01 12:58:51.341932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:51.341906 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0cd09f-58466d8c5-jkxqv"] Nov 01 12:58:52.369730 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:52.369699 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85d3fb7-bd4a-41f8-8274-3efda26c7f03" path="/var/lib/kubelet/pods/b85d3fb7-bd4a-41f8-8274-3efda26c7f03/volumes" Nov 01 12:58:54.004075 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:54.003987 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:57.197253 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:57.197222 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:57.329271 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:57.329241 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:57.452125 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:57.452046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:58.538893 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.538858 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28"] Nov 01 12:58:58.539284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.539056 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" podUID="663731f6-87fc-4254-ac7f-b5469d6643f7" containerName="maas-api" containerID="cri-o://88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed" gracePeriod=30 Nov 01 12:58:58.632421 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.632375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:58:58.666469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.666438 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" podUID="663731f6-87fc-4254-ac7f-b5469d6643f7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.114:8443/health\": dial tcp 10.133.0.114:8443: connect: connection refused" Nov 01 12:58:58.839657 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.839628 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:58:58.899079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.899052 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs\") pod \"663731f6-87fc-4254-ac7f-b5469d6643f7\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " Nov 01 12:58:58.899196 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.899108 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28\" (UniqueName: \"kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28\") pod \"663731f6-87fc-4254-ac7f-b5469d6643f7\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " Nov 01 12:58:58.899248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.899217 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca\") pod \"663731f6-87fc-4254-ac7f-b5469d6643f7\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " Nov 01 12:58:58.899407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.899248 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\") pod \"663731f6-87fc-4254-ac7f-b5469d6643f7\" (UID: \"663731f6-87fc-4254-ac7f-b5469d6643f7\") " Nov 01 12:58:58.899645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.899619 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca" pod "663731f6-87fc-4254-ac7f-b5469d6643f7" (UID: "663731f6-87fc-4254-ac7f-b5469d6643f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:58:58.899784 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.899768 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663731f6-87fc-4254-ac7f-b5469d6643f7-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:58.901239 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.901211 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs" pod "663731f6-87fc-4254-ac7f-b5469d6643f7" (UID: "663731f6-87fc-4254-ac7f-b5469d6643f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:58:58.901372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.901248 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls" pod "663731f6-87fc-4254-ac7f-b5469d6643f7" (UID: "663731f6-87fc-4254-ac7f-b5469d6643f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:58:58.901372 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:58.901271 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28" pod "663731f6-87fc-4254-ac7f-b5469d6643f7" (UID: "663731f6-87fc-4254-ac7f-b5469d6643f7"). InnerVolumeSpecName "kube-api-access-rdg28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:58:59.001584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.001500 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:59.001584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.001539 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663731f6-87fc-4254-ac7f-b5469d6643f7-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:59.001584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.001557 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdg28\" (UniqueName: \"kubernetes.io/projected/663731f6-87fc-4254-ac7f-b5469d6643f7-kube-api-access-rdg28\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:58:59.236505 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.236471 4484 generic.go:356] "Generic (PLEG): container finished" podID="663731f6-87fc-4254-ac7f-b5469d6643f7" containerID="88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed" exitCode=0 Nov 01 12:58:59.236697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.236525 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" Nov 01 12:58:59.236697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.236553 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" event={"ID":"663731f6-87fc-4254-ac7f-b5469d6643f7","Type":"ContainerDied","Data":"88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed"} Nov 01 12:58:59.236697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.236603 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28" event={"ID":"663731f6-87fc-4254-ac7f-b5469d6643f7","Type":"ContainerDied","Data":"1fb8777a8bcd0d13b8c44ea00c4612764b600811793f4bd9b7df6e42860e34f5"} Nov 01 12:58:59.236697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.236625 4484 scope.go:122] "RemoveContainer" containerID="88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed" Nov 01 12:58:59.247118 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.247100 4484 scope.go:122] "RemoveContainer" containerID="88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed" Nov 01 12:58:59.247392 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:58:59.247372 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed\": container with ID starting with 88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed not found: ID does not exist" containerID="88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed" Nov 01 12:58:59.247451 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.247397 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed"} err="failed to get container status \"88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed\": rpc error: code = NotFound desc = could not find container \"88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed\": container with ID starting with 88d3a1b56b06be3dac96c51acc6268c4c711e69cd48f2b8831a91332c6fbbeed not found: ID does not exist" Nov 01 12:58:59.390600 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.390570 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28"] Nov 01 12:58:59.436607 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.436577 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-c6x28"] Nov 01 12:58:59.835913 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.835881 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb"] Nov 01 12:58:59.836403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.836083 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" podUID="3dbc5032-84f7-40fd-876a-0805511d40aa" containerName="maas-api" containerID="cri-o://59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf" gracePeriod=30 Nov 01 12:58:59.922645 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:58:59.922619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:00.139012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.138989 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:59:00.216977 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.216950 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\") pod \"3dbc5032-84f7-40fd-876a-0805511d40aa\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " Nov 01 12:59:00.217142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.217132 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6\" (UniqueName: \"kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6\") pod \"3dbc5032-84f7-40fd-876a-0805511d40aa\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " Nov 01 12:59:00.217199 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.217184 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca\") pod \"3dbc5032-84f7-40fd-876a-0805511d40aa\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " Nov 01 12:59:00.217257 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.217206 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs\") pod \"3dbc5032-84f7-40fd-876a-0805511d40aa\" (UID: \"3dbc5032-84f7-40fd-876a-0805511d40aa\") " Nov 01 12:59:00.217600 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.217568 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca" pod "3dbc5032-84f7-40fd-876a-0805511d40aa" (UID: "3dbc5032-84f7-40fd-876a-0805511d40aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:00.219040 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.219019 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls" pod "3dbc5032-84f7-40fd-876a-0805511d40aa" (UID: "3dbc5032-84f7-40fd-876a-0805511d40aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:00.219207 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.219189 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs" pod "3dbc5032-84f7-40fd-876a-0805511d40aa" (UID: "3dbc5032-84f7-40fd-876a-0805511d40aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:00.219326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.219295 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6" pod "3dbc5032-84f7-40fd-876a-0805511d40aa" (UID: "3dbc5032-84f7-40fd-876a-0805511d40aa"). InnerVolumeSpecName "kube-api-access-svtw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:00.269481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.268873 4484 generic.go:356] "Generic (PLEG): container finished" podID="3dbc5032-84f7-40fd-876a-0805511d40aa" containerID="59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf" exitCode=0 Nov 01 12:59:00.269481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.269140 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" event={"ID":"3dbc5032-84f7-40fd-876a-0805511d40aa","Type":"ContainerDied","Data":"59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf"} Nov 01 12:59:00.269481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.269187 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" event={"ID":"3dbc5032-84f7-40fd-876a-0805511d40aa","Type":"ContainerDied","Data":"70ce08a30aaa3b88c6611dddbd25377740cee061a715f8eec66a229a8cd16b53"} Nov 01 12:59:00.269481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.269216 4484 scope.go:122] "RemoveContainer" containerID="59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf" Nov 01 12:59:00.269481 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.269349 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb" Nov 01 12:59:00.300000 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:00.298349 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:59:00.319140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.319112 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtw6\" (UniqueName: \"kubernetes.io/projected/3dbc5032-84f7-40fd-876a-0805511d40aa-kube-api-access-svtw6\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:00.319140 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.319140 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dbc5032-84f7-40fd-876a-0805511d40aa-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:00.319398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.319150 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:00.319398 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.319159 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dbc5032-84f7-40fd-876a-0805511d40aa-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:00.328741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.328716 4484 scope.go:122] "RemoveContainer" containerID="59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf" Nov 01 12:59:00.329102 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:00.329073 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf\": container with ID starting with 59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf not found: ID does not exist" containerID="59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf" Nov 01 12:59:00.329197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.329114 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf"} err="failed to get container status \"59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf\": rpc error: code = NotFound desc = could not find container \"59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf\": container with ID starting with 59fc2e22eb11c87c657b09b0b7fe705516c33bc0127f3874bca911330ebd4fbf not found: ID does not exist" Nov 01 12:59:00.336272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.336248 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:00.379750 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.372716 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663731f6-87fc-4254-ac7f-b5469d6643f7" path="/var/lib/kubelet/pods/663731f6-87fc-4254-ac7f-b5469d6643f7/volumes" Nov 01 12:59:00.452032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.451996 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb"] Nov 01 12:59:00.455155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.455131 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0a0610-8498c7b956-6kznb"] Nov 01 12:59:00.754541 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.750782 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:00.926963 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.926927 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr"] Nov 01 12:59:00.937490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:00.937456 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-998lr"] Nov 01 12:59:01.310643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:01.310613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:01.689606 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:01.689089 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:02.027460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.027426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:02.354829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.354731 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb"] Nov 01 12:59:02.369009 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.368981 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbc5032-84f7-40fd-876a-0805511d40aa" path="/var/lib/kubelet/pods/3dbc5032-84f7-40fd-876a-0805511d40aa/volumes" Nov 01 12:59:02.369446 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.369431 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d957e0-ef5c-4c66-a910-cde0deb006c4" path="/var/lib/kubelet/pods/72d957e0-ef5c-4c66-a910-cde0deb006c4/volumes" Nov 01 12:59:02.398868 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.398841 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0a0610-f6ssb"] Nov 01 12:59:02.637902 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.637832 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:02.753510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:02.753485 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:03.056483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:03.056455 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:03.407687 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:03.407612 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:04.028493 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:04.028457 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:04.369559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:04.369484 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7724384e-c25d-403c-8ac1-3986cd143dbf" path="/var/lib/kubelet/pods/7724384e-c25d-403c-8ac1-3986cd143dbf/volumes" Nov 01 12:59:04.525491 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:04.525457 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:04.774700 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:04.774675 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:04.895442 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:04.895407 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:05.147031 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:05.146942 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh"] Nov 01 12:59:05.192950 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:05.192913 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c7206f-sbhhh"] Nov 01 12:59:05.484828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:05.484800 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:06.369104 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:06.369071 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2" path="/var/lib/kubelet/pods/af0c1f43-a2f2-4c5f-9c0e-42de8e2b2dd2/volumes" Nov 01 12:59:06.555462 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:06.555433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:08.487698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:08.487670 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:08.613851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:08.613783 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:08.820335 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:08.820282 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd"] Nov 01 12:59:08.820568 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:08.820542 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" containerID="cri-o://3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8" gracePeriod=30 Nov 01 12:59:08.820830 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:08.820803 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:10.084047 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:10.083990 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:10.284526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:10.284489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:10.416067 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:10.415976 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:10.558326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:10.554346 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:10.744604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:10.744574 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:11.781822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:11.781741 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:11.935126 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:11.935086 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:12.109288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.109192 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:12.662508 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.662473 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:12.664197 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664168 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c"] Nov 01 12:59:12.664733 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664715 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3dbc5032-84f7-40fd-876a-0805511d40aa" containerName="maas-api" Nov 01 12:59:12.664822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664736 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbc5032-84f7-40fd-876a-0805511d40aa" containerName="maas-api" Nov 01 12:59:12.664822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664784 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e7564a4b-f381-46d0-9f27-8ad59e0a1050" containerName="revoke-keys" Nov 01 12:59:12.664822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664793 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7564a4b-f381-46d0-9f27-8ad59e0a1050" containerName="revoke-keys" Nov 01 12:59:12.664822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664819 4484 cpu_manager.go:414] "containerMap: removing container" podUID="663731f6-87fc-4254-ac7f-b5469d6643f7" containerName="maas-api" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664828 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663731f6-87fc-4254-ac7f-b5469d6643f7" containerName="maas-api" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664848 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b85d3fb7-bd4a-41f8-8274-3efda26c7f03" containerName="payload-pre-processing" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664856 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85d3fb7-bd4a-41f8-8274-3efda26c7f03" containerName="payload-pre-processing" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664870 4484 cpu_manager.go:414] "containerMap: removing container" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.664877 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baae4295-9113-41d7-a6db-1d030f4590e5" containerName="istio-proxy" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.665017 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7564a4b-f381-46d0-9f27-8ad59e0a1050" containerName="revoke-keys" Nov 01 12:59:12.665033 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.665034 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="663731f6-87fc-4254-ac7f-b5469d6643f7" containerName="maas-api" Nov 01 12:59:12.665440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.665045 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85d3fb7-bd4a-41f8-8274-3efda26c7f03" containerName="payload-pre-processing" Nov 01 12:59:12.665440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.665060 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbc5032-84f7-40fd-876a-0805511d40aa" containerName="maas-api" Nov 01 12:59:12.670141 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.670119 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.712355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.712323 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.712460 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.712373 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.712545 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.712480 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkh2b\" (UniqueName: \"kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.724329 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.724294 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c"] Nov 01 12:59:12.768828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.768681 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:59:12.768828 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.768788 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:59:12.815348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.813112 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:12.821355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.818558 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.821355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.818644 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.821355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.818737 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkh2b\" (UniqueName: \"kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.821355 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.820283 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.849326 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.840666 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.872658 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.872634 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkh2b\" (UniqueName: \"kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b\") pod \"maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:12.966392 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.961437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:12.986098 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:12.986067 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:13.071589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.071531 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw"] Nov 01 12:59:13.075986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.075872 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.112777 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.112748 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg"] Nov 01 12:59:13.116668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.116649 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.122971 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.122941 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zxc\" (UniqueName: \"kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.123057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.123012 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.123195 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.123168 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqkq\" (UniqueName: \"kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.123262 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.123234 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.123333 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.123318 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.124796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.124773 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-187d06ed\"" Nov 01 12:59:13.124796 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.124796 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-187d06ed-dockercfg-bt46g\"" Nov 01 12:59:13.175944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.175874 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw"] Nov 01 12:59:13.227459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.227409 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg"] Nov 01 12:59:13.227790 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.227570 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.228025 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.228002 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqkq\" (UniqueName: \"kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.228288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.228254 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.228680 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.228642 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.229012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.228984 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zxc\" (UniqueName: \"kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.229175 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.228410 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.230535 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.230507 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.239703 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.239661 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.249054 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.249032 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x"] Nov 01 12:59:13.253250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.253232 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.282149 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.282122 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqkq\" (UniqueName: \"kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.282274 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.282255 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zxc\" (UniqueName: \"kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.307853 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.307823 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x"] Nov 01 12:59:13.330720 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.330684 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.330931 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.330911 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwbp\" (UniqueName: \"kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.358892 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.358861 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c"] Nov 01 12:59:13.365369 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:13.365346 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce6085e_963c_453f_8acc_5d1f6bafc523.slice/crio-ae2127ab5794488700acbd6ecc64d34d3000ba53e13589d0f1e6f55034739856 WatchSource:0}: Error finding container ae2127ab5794488700acbd6ecc64d34d3000ba53e13589d0f1e6f55034739856: Status 404 returned error can't find the container with id ae2127ab5794488700acbd6ecc64d34d3000ba53e13589d0f1e6f55034739856 Nov 01 12:59:13.393534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.393513 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:13.396763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.396740 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" event={"ID":"7ce6085e-963c-453f-8acc-5d1f6bafc523","Type":"ContainerStarted","Data":"ae2127ab5794488700acbd6ecc64d34d3000ba53e13589d0f1e6f55034739856"} Nov 01 12:59:13.430735 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.430711 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:13.432598 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.432570 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.432809 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.432776 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwbp\" (UniqueName: \"kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.433458 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.433435 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.488926 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.488853 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwbp\" (UniqueName: \"kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.567526 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.567500 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:13.766432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.766410 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw"] Nov 01 12:59:13.813640 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:13.813612 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66ddcd7_a9fd_4d00_8aef_9da273265a62.slice/crio-4b4f2bb8723738ceacb49485521e184e5da9433eca0e55d15b177d51a6859463 WatchSource:0}: Error finding container 4b4f2bb8723738ceacb49485521e184e5da9433eca0e55d15b177d51a6859463: Status 404 returned error can't find the container with id 4b4f2bb8723738ceacb49485521e184e5da9433eca0e55d15b177d51a6859463 Nov 01 12:59:13.818821 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.818798 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg"] Nov 01 12:59:13.948247 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:13.948222 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x"] Nov 01 12:59:13.953137 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:13.953111 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e70166_b969_4d3d_baa1_dae474808bf3.slice/crio-3801ac8e46bfc9d618f5738f3bf46a5feaad32e59bbbc737ac54fb3f46114679 WatchSource:0}: Error finding container 3801ac8e46bfc9d618f5738f3bf46a5feaad32e59bbbc737ac54fb3f46114679: Status 404 returned error can't find the container with id 3801ac8e46bfc9d618f5738f3bf46a5feaad32e59bbbc737ac54fb3f46114679 Nov 01 12:59:14.152996 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.152970 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:59:14.242350 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242321 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242475 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242429 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242534 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242522 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242624 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242599 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242682 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242637 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242719 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242797 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242757 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242799 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert\") pod \"fd70cefc-cb18-4e20-a779-3f495184858d\" (UID: \"fd70cefc-cb18-4e20-a779-3f495184858d\") " Nov 01 12:59:14.242869 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.242836 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:14.243162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243023 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:14.243250 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243179 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:14.243348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243256 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:14.243404 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243375 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:14.243486 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243461 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.243616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243496 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.243616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243514 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.243616 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.243528 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.244901 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.244878 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 12:59:14.245348 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.245283 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "kube-api-access-5qjkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:14.245434 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.245369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:14.245479 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.245447 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy" pod "fd70cefc-cb18-4e20-a779-3f495184858d" (UID: "fd70cefc-cb18-4e20-a779-3f495184858d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:14.344259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.344185 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd70cefc-cb18-4e20-a779-3f495184858d-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.344259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.344212 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd70cefc-cb18-4e20-a779-3f495184858d-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.344259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.344220 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.344259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.344229 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd70cefc-cb18-4e20-a779-3f495184858d-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.344259 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.344238 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjkq\" (UniqueName: \"kubernetes.io/projected/fd70cefc-cb18-4e20-a779-3f495184858d-kube-api-access-5qjkq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:14.403841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.403810 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" event={"ID":"d33b08c3-1d24-4659-a46f-70587947cd17","Type":"ContainerStarted","Data":"343a1d6c2e2ab3730706acc09b7d050663f20fda90021532c3e2d85c8f3a89b7"} Nov 01 12:59:14.403989 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.403871 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" event={"ID":"d33b08c3-1d24-4659-a46f-70587947cd17","Type":"ContainerStarted","Data":"25045e0f2649ef4567aa8219b7a672077227c98379dd2d8aef93f8a6485fb959"} Nov 01 12:59:14.405374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.405344 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" event={"ID":"e5e70166-b969-4d3d-baa1-dae474808bf3","Type":"ContainerStarted","Data":"6ce0fa0964ad702450f49f898072fc910b6460e726416bd21323af7982a6c02c"} Nov 01 12:59:14.405490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.405400 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" event={"ID":"e5e70166-b969-4d3d-baa1-dae474808bf3","Type":"ContainerStarted","Data":"3801ac8e46bfc9d618f5738f3bf46a5feaad32e59bbbc737ac54fb3f46114679"} Nov 01 12:59:14.405490 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.405432 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:14.407008 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.406984 4484 generic.go:356] "Generic (PLEG): container finished" podID="fd70cefc-cb18-4e20-a779-3f495184858d" containerID="3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8" exitCode=0 Nov 01 12:59:14.407093 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.407022 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" Nov 01 12:59:14.407093 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.407072 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" event={"ID":"fd70cefc-cb18-4e20-a779-3f495184858d","Type":"ContainerDied","Data":"3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8"} Nov 01 12:59:14.407214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.407121 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd" event={"ID":"fd70cefc-cb18-4e20-a779-3f495184858d","Type":"ContainerDied","Data":"0f0c0759c7691d37802f40454c06d5fc52bb0e829504d8ac07d2827868765733"} Nov 01 12:59:14.407214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.407155 4484 scope.go:122] "RemoveContainer" containerID="3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8" Nov 01 12:59:14.408925 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.408900 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" event={"ID":"7ce6085e-963c-453f-8acc-5d1f6bafc523","Type":"ContainerStarted","Data":"4a91890dfbc18a0793f0d4a289eb00b196042ddf3d13df8d05a80687792c86a2"} Nov 01 12:59:14.410822 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.410802 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" event={"ID":"c66ddcd7-a9fd-4d00-8aef-9da273265a62","Type":"ContainerStarted","Data":"dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a"} Nov 01 12:59:14.410921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.410835 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" event={"ID":"c66ddcd7-a9fd-4d00-8aef-9da273265a62","Type":"ContainerStarted","Data":"4b4f2bb8723738ceacb49485521e184e5da9433eca0e55d15b177d51a6859463"} Nov 01 12:59:14.410921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.410897 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:14.417411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.417391 4484 scope.go:122] "RemoveContainer" containerID="6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21" Nov 01 12:59:14.473746 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.473725 4484 scope.go:122] "RemoveContainer" containerID="3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8" Nov 01 12:59:14.474056 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:14.474034 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8\": container with ID starting with 3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8 not found: ID does not exist" containerID="3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8" Nov 01 12:59:14.474138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.474064 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8"} err="failed to get container status \"3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8\": rpc error: code = NotFound desc = could not find container \"3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8\": container with ID starting with 3b9362cd1ca3acc2101033d6718269c425cdea0f86809d5d8808395f68d1f2d8 not found: ID does not exist" Nov 01 12:59:14.474138 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.474083 4484 scope.go:122] "RemoveContainer" containerID="6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21" Nov 01 12:59:14.474391 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:14.474370 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21\": container with ID starting with 6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21 not found: ID does not exist" containerID="6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21" Nov 01 12:59:14.474455 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.474402 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21"} err="failed to get container status \"6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21\": rpc error: code = NotFound desc = could not find container \"6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21\": container with ID starting with 6510bf34d071397291d37dee476b4ff82d9795925ed643fded750fec93775e21 not found: ID does not exist" Nov 01 12:59:14.513827 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.513783 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" podStartSLOduration=1.513770288 podStartE2EDuration="1.513770288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:14.511000218 +0000 UTC m=+2224.260943194" watchObservedRunningTime="2026-11-01 12:59:14.513770288 +0000 UTC m=+2224.263713265" Nov 01 12:59:14.618403 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.618316 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" podStartSLOduration=2.618286413 podStartE2EDuration="2.618286413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:14.616974793 +0000 UTC m=+2224.366917771" watchObservedRunningTime="2026-11-01 12:59:14.618286413 +0000 UTC m=+2224.368229411" Nov 01 12:59:14.726162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.726113 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" podStartSLOduration=1.726099697 podStartE2EDuration="1.726099697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:14.724337778 +0000 UTC m=+2224.474280753" watchObservedRunningTime="2026-11-01 12:59:14.726099697 +0000 UTC m=+2224.476042675" Nov 01 12:59:14.833763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.833616 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd"] Nov 01 12:59:14.836480 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.836445 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0a0610-openshift-default-ddfb9c9f5-d8tcd"] Nov 01 12:59:14.942631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:14.942530 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" podStartSLOduration=1.9425168080000002 podStartE2EDuration="1.942516808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:14.939340252 +0000 UTC m=+2224.689283229" watchObservedRunningTime="2026-11-01 12:59:14.942516808 +0000 UTC m=+2224.692459785" Nov 01 12:59:15.418851 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.418820 4484 generic.go:356] "Generic (PLEG): container finished" podID="d33b08c3-1d24-4659-a46f-70587947cd17" containerID="343a1d6c2e2ab3730706acc09b7d050663f20fda90021532c3e2d85c8f3a89b7" exitCode=0 Nov 01 12:59:15.419057 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.418908 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" event={"ID":"d33b08c3-1d24-4659-a46f-70587947cd17","Type":"ContainerDied","Data":"343a1d6c2e2ab3730706acc09b7d050663f20fda90021532c3e2d85c8f3a89b7"} Nov 01 12:59:15.421359 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.421331 4484 generic.go:356] "Generic (PLEG): container finished" podID="7ce6085e-963c-453f-8acc-5d1f6bafc523" containerID="4a91890dfbc18a0793f0d4a289eb00b196042ddf3d13df8d05a80687792c86a2" exitCode=0 Nov 01 12:59:15.421483 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.421405 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" event={"ID":"7ce6085e-963c-453f-8acc-5d1f6bafc523","Type":"ContainerDied","Data":"4a91890dfbc18a0793f0d4a289eb00b196042ddf3d13df8d05a80687792c86a2"} Nov 01 12:59:15.473142 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.473122 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp"] Nov 01 12:59:15.519137 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.519113 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr"] Nov 01 12:59:15.590288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:15.590259 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:16.369528 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.369497 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" path="/var/lib/kubelet/pods/fd70cefc-cb18-4e20-a779-3f495184858d/volumes" Nov 01 12:59:16.575249 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.575220 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:16.607513 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.607493 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:16.693715 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.693638 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca\") pod \"d33b08c3-1d24-4659-a46f-70587947cd17\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " Nov 01 12:59:16.693852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.693726 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq\" (UniqueName: \"kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq\") pod \"d33b08c3-1d24-4659-a46f-70587947cd17\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " Nov 01 12:59:16.693852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.693795 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca\") pod \"7ce6085e-963c-453f-8acc-5d1f6bafc523\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " Nov 01 12:59:16.693852 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.693828 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca\") pod \"d33b08c3-1d24-4659-a46f-70587947cd17\" (UID: \"d33b08c3-1d24-4659-a46f-70587947cd17\") " Nov 01 12:59:16.694040 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.693897 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b\" (UniqueName: \"kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b\") pod \"7ce6085e-963c-453f-8acc-5d1f6bafc523\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " Nov 01 12:59:16.694040 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.693965 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca\") pod \"7ce6085e-963c-453f-8acc-5d1f6bafc523\" (UID: \"7ce6085e-963c-453f-8acc-5d1f6bafc523\") " Nov 01 12:59:16.694264 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.694235 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca" pod "d33b08c3-1d24-4659-a46f-70587947cd17" (UID: "d33b08c3-1d24-4659-a46f-70587947cd17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:16.694681 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.694651 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca" pod "7ce6085e-963c-453f-8acc-5d1f6bafc523" (UID: "7ce6085e-963c-453f-8acc-5d1f6bafc523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:16.695012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.694976 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:16.695012 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.695005 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:16.696914 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.696886 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq" pod "d33b08c3-1d24-4659-a46f-70587947cd17" (UID: "d33b08c3-1d24-4659-a46f-70587947cd17"). InnerVolumeSpecName "kube-api-access-5zqkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:16.697020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.696989 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca" pod "7ce6085e-963c-453f-8acc-5d1f6bafc523" (UID: "7ce6085e-963c-453f-8acc-5d1f6bafc523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:16.697020 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.697004 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca" pod "d33b08c3-1d24-4659-a46f-70587947cd17" (UID: "d33b08c3-1d24-4659-a46f-70587947cd17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:16.697146 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.697126 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b" pod "7ce6085e-963c-453f-8acc-5d1f6bafc523" (UID: "7ce6085e-963c-453f-8acc-5d1f6bafc523"). InnerVolumeSpecName "kube-api-access-nkh2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:16.796123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.796092 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b08c3-1d24-4659-a46f-70587947cd17-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:16.796123 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.796122 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zqkq\" (UniqueName: \"kubernetes.io/projected/d33b08c3-1d24-4659-a46f-70587947cd17-kube-api-access-5zqkq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:16.796293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.796134 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce6085e-963c-453f-8acc-5d1f6bafc523-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:16.796293 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.796143 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkh2b\" (UniqueName: \"kubernetes.io/projected/7ce6085e-963c-453f-8acc-5d1f6bafc523-kube-api-access-nkh2b\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:16.796661 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:16.796506 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:17.232697 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.232664 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 12:59:17.434554 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.434525 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" Nov 01 12:59:17.434987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.434555 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg" event={"ID":"d33b08c3-1d24-4659-a46f-70587947cd17","Type":"ContainerDied","Data":"25045e0f2649ef4567aa8219b7a672077227c98379dd2d8aef93f8a6485fb959"} Nov 01 12:59:17.434987 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.434604 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25045e0f2649ef4567aa8219b7a672077227c98379dd2d8aef93f8a6485fb959" Nov 01 12:59:17.436000 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.435986 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" Nov 01 12:59:17.436077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.436034 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c" event={"ID":"7ce6085e-963c-453f-8acc-5d1f6bafc523","Type":"ContainerDied","Data":"ae2127ab5794488700acbd6ecc64d34d3000ba53e13589d0f1e6f55034739856"} Nov 01 12:59:17.436077 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:17.436073 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae2127ab5794488700acbd6ecc64d34d3000ba53e13589d0f1e6f55034739856" Nov 01 12:59:22.707381 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:22.707348 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:24.231223 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:24.231188 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph"] Nov 01 12:59:24.345642 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:24.345610 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm"] Nov 01 12:59:24.581204 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:24.581108 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:24.708272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:24.708242 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:25.424441 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:25.424413 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 12:59:25.424814 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:25.424477 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 12:59:26.664470 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:26.664436 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr"] Nov 01 12:59:27.083704 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:27.083674 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:27.201014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:27.200985 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:27.850391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:27.850331 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:29.232440 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.232400 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" podUID="60e1c795-52e6-4891-bc9c-4d696f87dfdf" containerName="payload-processing" containerID="cri-o://c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539" gracePeriod=25 Nov 01 12:59:29.346501 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.346436 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" podUID="cc4552e6-6af3-4009-8691-82f08f571deb" containerName="payload-pre-processing" containerID="cri-o://2d3f684e4397769e64121f5f5a58122bba5bb8c35b4af3b62b7a393eef2b8f0b" gracePeriod=25 Nov 01 12:59:29.500011 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.499945 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:59:29.517375 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.517341 4484 generic.go:356] "Generic (PLEG): container finished" podID="cc4552e6-6af3-4009-8691-82f08f571deb" containerID="2d3f684e4397769e64121f5f5a58122bba5bb8c35b4af3b62b7a393eef2b8f0b" exitCode=0 Nov 01 12:59:29.517494 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.517412 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" event={"ID":"cc4552e6-6af3-4009-8691-82f08f571deb","Type":"ContainerDied","Data":"2d3f684e4397769e64121f5f5a58122bba5bb8c35b4af3b62b7a393eef2b8f0b"} Nov 01 12:59:29.519659 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.519629 4484 generic.go:356] "Generic (PLEG): container finished" podID="60e1c795-52e6-4891-bc9c-4d696f87dfdf" containerID="c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539" exitCode=0 Nov 01 12:59:29.519862 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.519843 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" event={"ID":"60e1c795-52e6-4891-bc9c-4d696f87dfdf","Type":"ContainerDied","Data":"c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539"} Nov 01 12:59:29.519945 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.519894 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" event={"ID":"60e1c795-52e6-4891-bc9c-4d696f87dfdf","Type":"ContainerDied","Data":"62ae076c286192204bcdbc9fa1cfccf51071bf9ee4a5dd9540b626d498524e79"} Nov 01 12:59:29.520007 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.519927 4484 scope.go:122] "RemoveContainer" containerID="c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539" Nov 01 12:59:29.520105 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.520090 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph" Nov 01 12:59:29.533807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.533786 4484 scope.go:122] "RemoveContainer" containerID="c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539" Nov 01 12:59:29.534084 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:29.534063 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539\": container with ID starting with c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539 not found: ID does not exist" containerID="c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539" Nov 01 12:59:29.534159 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.534095 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539"} err="failed to get container status \"c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539\": rpc error: code = NotFound desc = could not find container \"c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539\": container with ID starting with c3dc458c26fd14103e375e759b797fe90d84ede505eb4c8396d8e847f2e09539 not found: ID does not exist" Nov 01 12:59:29.616504 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.616479 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:59:29.658471 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.658442 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz\" (UniqueName: \"kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz\") pod \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " Nov 01 12:59:29.658670 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.658512 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume\") pod \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\" (UID: \"60e1c795-52e6-4891-bc9c-4d696f87dfdf\") " Nov 01 12:59:29.658986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.658958 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume" pod "60e1c795-52e6-4891-bc9c-4d696f87dfdf" (UID: "60e1c795-52e6-4891-bc9c-4d696f87dfdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:29.660546 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.660518 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz" pod "60e1c795-52e6-4891-bc9c-4d696f87dfdf" (UID: "60e1c795-52e6-4891-bc9c-4d696f87dfdf"). InnerVolumeSpecName "kube-api-access-rsvqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:29.759411 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.759337 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt\" (UniqueName: \"kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt\") pod \"cc4552e6-6af3-4009-8691-82f08f571deb\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " Nov 01 12:59:29.759571 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.759428 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume\") pod \"cc4552e6-6af3-4009-8691-82f08f571deb\" (UID: \"cc4552e6-6af3-4009-8691-82f08f571deb\") " Nov 01 12:59:29.759810 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.759779 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume" pod "cc4552e6-6af3-4009-8691-82f08f571deb" (UID: "cc4552e6-6af3-4009-8691-82f08f571deb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:29.760085 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.760065 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvqz\" (UniqueName: \"kubernetes.io/projected/60e1c795-52e6-4891-bc9c-4d696f87dfdf-kube-api-access-rsvqz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:29.760165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.760088 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e1c795-52e6-4891-bc9c-4d696f87dfdf-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:29.760165 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.760099 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4552e6-6af3-4009-8691-82f08f571deb-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:29.761935 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.761915 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt" pod "cc4552e6-6af3-4009-8691-82f08f571deb" (UID: "cc4552e6-6af3-4009-8691-82f08f571deb"). InnerVolumeSpecName "kube-api-access-dk2qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:29.861550 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.861529 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2qt\" (UniqueName: \"kubernetes.io/projected/cc4552e6-6af3-4009-8691-82f08f571deb-kube-api-access-dk2qt\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:29.934151 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.934121 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph"] Nov 01 12:59:29.937101 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:29.937080 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78c69d-86c5dc4cd9-qngph"] Nov 01 12:59:30.251112 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:30.251082 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 12:59:30.370120 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.370088 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e1c795-52e6-4891-bc9c-4d696f87dfdf" path="/var/lib/kubelet/pods/60e1c795-52e6-4891-bc9c-4d696f87dfdf/volumes" Nov 01 12:59:30.493668 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.493629 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" containerID="cri-o://2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b" gracePeriod=45 Nov 01 12:59:30.527231 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.527206 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" Nov 01 12:59:30.527231 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.527215 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm" event={"ID":"cc4552e6-6af3-4009-8691-82f08f571deb","Type":"ContainerDied","Data":"a9900a764118b92961cfcaa0f2528f383cde610a2d09401ef16fe879fc846291"} Nov 01 12:59:30.527415 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.527271 4484 scope.go:122] "RemoveContainer" containerID="2d3f684e4397769e64121f5f5a58122bba5bb8c35b4af3b62b7a393eef2b8f0b" Nov 01 12:59:30.542213 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.542186 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" containerID="cri-o://3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d" gracePeriod=45 Nov 01 12:59:30.738829 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.738654 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm"] Nov 01 12:59:30.784968 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.784937 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c69d-7fbfd6dc55-4tksm"] Nov 01 12:59:30.797453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.797432 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:59:30.817038 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.817020 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:59:30.974752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.974722 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs\") pod \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " Nov 01 12:59:30.974942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.974766 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm\") pod \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " Nov 01 12:59:30.974942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.974795 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache\") pod \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " Nov 01 12:59:30.974942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.974846 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home\") pod \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " Nov 01 12:59:30.974942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.974894 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp\" (UniqueName: \"kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp\") pod \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " Nov 01 12:59:30.974942 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.974920 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm\") pod \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " Nov 01 12:59:30.975252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975006 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache\") pod \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " Nov 01 12:59:30.975252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975045 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home\") pod \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " Nov 01 12:59:30.975252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975099 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs\") pod \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " Nov 01 12:59:30.975252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975135 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9\" (UniqueName: \"kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9\") pod \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " Nov 01 12:59:30.975252 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975213 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir\") pod \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\" (UID: \"e7f1ee28-a99a-4dea-9890-4474f8c0e312\") " Nov 01 12:59:30.975555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975270 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir\") pod \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\" (UID: \"e52f59e0-1df0-41f0-b363-bb5962bd92a1\") " Nov 01 12:59:30.975555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache" pod "e7f1ee28-a99a-4dea-9890-4474f8c0e312" (UID: "e7f1ee28-a99a-4dea-9890-4474f8c0e312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.975676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975544 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home" pod "e52f59e0-1df0-41f0-b363-bb5962bd92a1" (UID: "e52f59e0-1df0-41f0-b363-bb5962bd92a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.975787 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975764 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir" pod "e52f59e0-1df0-41f0-b363-bb5962bd92a1" (UID: "e52f59e0-1df0-41f0-b363-bb5962bd92a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.975964 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.975940 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home" pod "e7f1ee28-a99a-4dea-9890-4474f8c0e312" (UID: "e7f1ee28-a99a-4dea-9890-4474f8c0e312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.976079 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.976055 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:30.976148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.976088 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:30.976148 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.976105 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:30.976671 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.976472 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache" pod "e52f59e0-1df0-41f0-b363-bb5962bd92a1" (UID: "e52f59e0-1df0-41f0-b363-bb5962bd92a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.976743 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.976716 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir" pod "e7f1ee28-a99a-4dea-9890-4474f8c0e312" (UID: "e7f1ee28-a99a-4dea-9890-4474f8c0e312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.978701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.978672 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs" pod "e52f59e0-1df0-41f0-b363-bb5962bd92a1" (UID: "e52f59e0-1df0-41f0-b363-bb5962bd92a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:30.978807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.978709 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm" pod "e7f1ee28-a99a-4dea-9890-4474f8c0e312" (UID: "e7f1ee28-a99a-4dea-9890-4474f8c0e312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.978807 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.978740 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm" pod "e52f59e0-1df0-41f0-b363-bb5962bd92a1" (UID: "e52f59e0-1df0-41f0-b363-bb5962bd92a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 12:59:30.978930 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.978834 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp" pod "e7f1ee28-a99a-4dea-9890-4474f8c0e312" (UID: "e7f1ee28-a99a-4dea-9890-4474f8c0e312"). InnerVolumeSpecName "kube-api-access-gbsvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:30.979155 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.979128 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs" pod "e7f1ee28-a99a-4dea-9890-4474f8c0e312" (UID: "e7f1ee28-a99a-4dea-9890-4474f8c0e312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:30.980666 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:30.980645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9" pod "e52f59e0-1df0-41f0-b363-bb5962bd92a1" (UID: "e52f59e0-1df0-41f0-b363-bb5962bd92a1"). InnerVolumeSpecName "kube-api-access-bzhb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:31.077143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077069 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077103 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f59e0-1df0-41f0-b363-bb5962bd92a1-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077118 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzhb9\" (UniqueName: \"kubernetes.io/projected/e52f59e0-1df0-41f0-b363-bb5962bd92a1-kube-api-access-bzhb9\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077143 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077134 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077147 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f1ee28-a99a-4dea-9890-4474f8c0e312-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077160 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f1ee28-a99a-4dea-9890-4474f8c0e312-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077174 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077187 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbsvp\" (UniqueName: \"kubernetes.io/projected/e7f1ee28-a99a-4dea-9890-4474f8c0e312-kube-api-access-gbsvp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.077416 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.077201 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f59e0-1df0-41f0-b363-bb5962bd92a1-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:31.536238 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.536208 4484 generic.go:356] "Generic (PLEG): container finished" podID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerID="2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b" exitCode=0 Nov 01 12:59:31.536722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.536253 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" Nov 01 12:59:31.536722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.536280 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" event={"ID":"e7f1ee28-a99a-4dea-9890-4474f8c0e312","Type":"ContainerDied","Data":"2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b"} Nov 01 12:59:31.536722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.536335 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp" event={"ID":"e7f1ee28-a99a-4dea-9890-4474f8c0e312","Type":"ContainerDied","Data":"0267017e2fe90ba20493a5682b71fee8def00a60c06dd268dbacfb029bd44898"} Nov 01 12:59:31.536722 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.536357 4484 scope.go:122] "RemoveContainer" containerID="2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b" Nov 01 12:59:31.539476 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.539451 4484 generic.go:356] "Generic (PLEG): container finished" podID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerID="3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d" exitCode=0 Nov 01 12:59:31.539608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.539489 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" Nov 01 12:59:31.539608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.539512 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" event={"ID":"e52f59e0-1df0-41f0-b363-bb5962bd92a1","Type":"ContainerDied","Data":"3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d"} Nov 01 12:59:31.539608 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.539549 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr" event={"ID":"e52f59e0-1df0-41f0-b363-bb5962bd92a1","Type":"ContainerDied","Data":"9928ac030ab329c79c65265ca9c99d4009c0ecc6ec47976c40c8d06316df5c08"} Nov 01 12:59:31.549811 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.549797 4484 scope.go:122] "RemoveContainer" containerID="2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b" Nov 01 12:59:31.550048 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:31.550029 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b\": container with ID starting with 2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b not found: ID does not exist" containerID="2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b" Nov 01 12:59:31.550093 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.550055 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b"} err="failed to get container status \"2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b\": rpc error: code = NotFound desc = could not find container \"2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b\": container with ID starting with 2009e6c70812718608ef221beda1e948736e90091d42a34ed20bb54119be729b not found: ID does not exist" Nov 01 12:59:31.550093 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.550070 4484 scope.go:122] "RemoveContainer" containerID="3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d" Nov 01 12:59:31.558514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.558501 4484 scope.go:122] "RemoveContainer" containerID="3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d" Nov 01 12:59:31.558759 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:31.558743 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d\": container with ID starting with 3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d not found: ID does not exist" containerID="3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d" Nov 01 12:59:31.558812 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.558764 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d"} err="failed to get container status \"3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d\": rpc error: code = NotFound desc = could not find container \"3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d\": container with ID starting with 3441649e4666d57b916d95d4720d91081c497295e3befc2137cea5259b67e68d not found: ID does not exist" Nov 01 12:59:31.665614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.665575 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerName="payload-processing" containerID="cri-o://e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4" gracePeriod=25 Nov 01 12:59:31.714327 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.714291 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr"] Nov 01 12:59:31.759072 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.759048 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/simulator-w0-94c94b-kserve-fc68b7f78-lfgkr"] Nov 01 12:59:31.817974 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.817903 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp"] Nov 01 12:59:31.822314 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:31.822269 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-w0-94c94b-kserve-544947d9fd-tczrp"] Nov 01 12:59:32.369280 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:32.369249 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4552e6-6af3-4009-8691-82f08f571deb" path="/var/lib/kubelet/pods/cc4552e6-6af3-4009-8691-82f08f571deb/volumes" Nov 01 12:59:32.369759 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:32.369742 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" path="/var/lib/kubelet/pods/e52f59e0-1df0-41f0-b363-bb5962bd92a1/volumes" Nov 01 12:59:32.370202 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:32.370188 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" path="/var/lib/kubelet/pods/e7f1ee28-a99a-4dea-9890-4474f8c0e312/volumes" Nov 01 12:59:33.207391 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:33.207353 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:38.329125 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:38.329084 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 12:59:38.702610 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:38.702536 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:43.367344 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:43.367272 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:44.714755 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.714717 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8"] Nov 01 12:59:44.715430 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715409 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cc4552e6-6af3-4009-8691-82f08f571deb" containerName="payload-pre-processing" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715432 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4552e6-6af3-4009-8691-82f08f571deb" containerName="payload-pre-processing" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715451 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d33b08c3-1d24-4659-a46f-70587947cd17" containerName="revoke-keys" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715459 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33b08c3-1d24-4659-a46f-70587947cd17" containerName="revoke-keys" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715470 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715477 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715505 4484 cpu_manager.go:414] "containerMap: removing container" podUID="60e1c795-52e6-4891-bc9c-4d696f87dfdf" containerName="payload-processing" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715513 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e1c795-52e6-4891-bc9c-4d696f87dfdf" containerName="payload-processing" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715534 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" Nov 01 12:59:44.715555 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715547 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715562 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7ce6085e-963c-453f-8acc-5d1f6bafc523" containerName="revoke-keys" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715569 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce6085e-963c-453f-8acc-5d1f6bafc523" containerName="revoke-keys" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715595 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715603 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715618 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715625 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715755 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715766 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52f59e0-1df0-41f0-b363-bb5962bd92a1" containerName="main" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715779 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4552e6-6af3-4009-8691-82f08f571deb" containerName="payload-pre-processing" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715788 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33b08c3-1d24-4659-a46f-70587947cd17" containerName="revoke-keys" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715797 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f1ee28-a99a-4dea-9890-4474f8c0e312" containerName="main" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715810 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e1c795-52e6-4891-bc9c-4d696f87dfdf" containerName="payload-processing" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715821 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd70cefc-cb18-4e20-a779-3f495184858d" containerName="istio-proxy" Nov 01 12:59:44.716014 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.715830 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce6085e-963c-453f-8acc-5d1f6bafc523" containerName="revoke-keys" Nov 01 12:59:44.719647 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.719624 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.817248 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.817222 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:59:44.817478 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.817459 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:59:44.820328 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.820295 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8"] Nov 01 12:59:44.822979 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.822954 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5"] Nov 01 12:59:44.826896 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.826878 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:44.832818 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.832792 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2stq\" (UniqueName: \"kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.832921 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.832854 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.832986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.832956 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.870170 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.870138 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5"] Nov 01 12:59:44.934584 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.934557 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:44.934752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.934685 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2stq\" (UniqueName: \"kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.934752 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.934732 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.934841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.934751 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zsj\" (UniqueName: \"kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:44.934841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.934789 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:44.934841 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.934814 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.935517 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.935494 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.937056 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.937036 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:44.988107 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:44.988085 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2stq\" (UniqueName: \"kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq\") pod \"maas-api-revoke-sub-96e1245f-7dbp8\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:45.033676 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.033651 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:45.036181 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.036159 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.036272 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.036259 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zsj\" (UniqueName: \"kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.036334 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.036289 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.037214 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.036905 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.038482 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.038458 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.090367 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.090275 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zsj\" (UniqueName: \"kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj\") pod \"maas-api-revoke-sub-42877763-xqrz5\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.140669 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.140642 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:45.395017 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.394945 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:45.405212 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.405186 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8"] Nov 01 12:59:45.410792 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:45.410770 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a624144_2e97_47c7_b648_3129e4d1c821.slice/crio-0220b08ffb992cd2c56ea7708ed89043ea70497a843ab9fd5a61b3b8d726221c WatchSource:0}: Error finding container 0220b08ffb992cd2c56ea7708ed89043ea70497a843ab9fd5a61b3b8d726221c: Status 404 returned error can't find the container with id 0220b08ffb992cd2c56ea7708ed89043ea70497a843ab9fd5a61b3b8d726221c Nov 01 12:59:45.511407 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.511383 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5"] Nov 01 12:59:45.517390 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:45.517362 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a3201b_a468_41ae_b865_8913bf516c11.slice/crio-c84f0da22bf302129075ac121d6ac09e63fecfd124c159e087ce7cbdff675d29 WatchSource:0}: Error finding container c84f0da22bf302129075ac121d6ac09e63fecfd124c159e087ce7cbdff675d29: Status 404 returned error can't find the container with id c84f0da22bf302129075ac121d6ac09e63fecfd124c159e087ce7cbdff675d29 Nov 01 12:59:45.592444 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.592412 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:45.624628 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.624603 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" event={"ID":"b0a3201b-a468-41ae-b865-8913bf516c11","Type":"ContainerStarted","Data":"c84f0da22bf302129075ac121d6ac09e63fecfd124c159e087ce7cbdff675d29"} Nov 01 12:59:45.629151 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.626574 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" event={"ID":"6a624144-2e97-47c7-b648-3129e4d1c821","Type":"ContainerStarted","Data":"0220b08ffb992cd2c56ea7708ed89043ea70497a843ab9fd5a61b3b8d726221c"} Nov 01 12:59:45.751868 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:45.751833 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:46.018980 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.018943 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk"] Nov 01 12:59:46.021604 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.021581 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0cd09f-p28rk"] Nov 01 12:59:46.340585 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.340553 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:46.369587 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.369556 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7564a4b-f381-46d0-9f27-8ad59e0a1050" path="/var/lib/kubelet/pods/e7564a4b-f381-46d0-9f27-8ad59e0a1050/volumes" Nov 01 12:59:46.514260 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.514233 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:46.634374 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.634342 4484 generic.go:356] "Generic (PLEG): container finished" podID="6a624144-2e97-47c7-b648-3129e4d1c821" containerID="5f90cc85f95590f9b3aca33d9d6a4db9ac8d7882e0d86f2b00313e6e97b1386e" exitCode=0 Nov 01 12:59:46.634552 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.634426 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" event={"ID":"6a624144-2e97-47c7-b648-3129e4d1c821","Type":"ContainerDied","Data":"5f90cc85f95590f9b3aca33d9d6a4db9ac8d7882e0d86f2b00313e6e97b1386e"} Nov 01 12:59:46.636051 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.636026 4484 generic.go:356] "Generic (PLEG): container finished" podID="b0a3201b-a468-41ae-b865-8913bf516c11" containerID="cb5ab2f09c336edab4cb6526361f3d006ec6adba75fb453c17f75acf99701bf3" exitCode=0 Nov 01 12:59:46.636162 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.636076 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" event={"ID":"b0a3201b-a468-41ae-b865-8913bf516c11","Type":"ContainerDied","Data":"cb5ab2f09c336edab4cb6526361f3d006ec6adba75fb453c17f75acf99701bf3"} Nov 01 12:59:46.643453 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:46.643433 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:47.779677 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.779653 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:47.810562 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.810541 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:47.866166 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.866136 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca\") pod \"b0a3201b-a468-41ae-b865-8913bf516c11\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " Nov 01 12:59:47.866323 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.866234 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca\") pod \"b0a3201b-a468-41ae-b865-8913bf516c11\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " Nov 01 12:59:47.866400 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.866323 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj\" (UniqueName: \"kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj\") pod \"b0a3201b-a468-41ae-b865-8913bf516c11\" (UID: \"b0a3201b-a468-41ae-b865-8913bf516c11\") " Nov 01 12:59:47.866553 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.866524 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca" pod "b0a3201b-a468-41ae-b865-8913bf516c11" (UID: "b0a3201b-a468-41ae-b865-8913bf516c11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:47.866766 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.866749 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:47.868469 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.868445 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj" pod "b0a3201b-a468-41ae-b865-8913bf516c11" (UID: "b0a3201b-a468-41ae-b865-8913bf516c11"). InnerVolumeSpecName "kube-api-access-78zsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:47.868559 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.868439 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca" pod "b0a3201b-a468-41ae-b865-8913bf516c11" (UID: "b0a3201b-a468-41ae-b865-8913bf516c11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:47.967763 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.967684 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca\") pod \"6a624144-2e97-47c7-b648-3129e4d1c821\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " Nov 01 12:59:47.967932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.967790 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca\") pod \"6a624144-2e97-47c7-b648-3129e4d1c821\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " Nov 01 12:59:47.967932 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.967878 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq\" (UniqueName: \"kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq\") pod \"6a624144-2e97-47c7-b648-3129e4d1c821\" (UID: \"6a624144-2e97-47c7-b648-3129e4d1c821\") " Nov 01 12:59:47.968139 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.968113 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca" pod "6a624144-2e97-47c7-b648-3129e4d1c821" (UID: "6a624144-2e97-47c7-b648-3129e4d1c821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:47.968284 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.968267 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78zsj\" (UniqueName: \"kubernetes.io/projected/b0a3201b-a468-41ae-b865-8913bf516c11-kube-api-access-78zsj\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:47.968369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.968285 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:47.968369 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.968295 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a3201b-a468-41ae-b865-8913bf516c11-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:47.969922 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.969904 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq" pod "6a624144-2e97-47c7-b648-3129e4d1c821" (UID: "6a624144-2e97-47c7-b648-3129e4d1c821"). InnerVolumeSpecName "kube-api-access-x2stq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:47.969992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:47.969948 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca" pod "6a624144-2e97-47c7-b648-3129e4d1c821" (UID: "6a624144-2e97-47c7-b648-3129e4d1c821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:48.069255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.069227 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a624144-2e97-47c7-b648-3129e4d1c821-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:48.069255 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.069253 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2stq\" (UniqueName: \"kubernetes.io/projected/6a624144-2e97-47c7-b648-3129e4d1c821-kube-api-access-x2stq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:48.328459 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.328425 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 12:59:48.649590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.649503 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" Nov 01 12:59:48.649590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.649541 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5" event={"ID":"b0a3201b-a468-41ae-b865-8913bf516c11","Type":"ContainerDied","Data":"c84f0da22bf302129075ac121d6ac09e63fecfd124c159e087ce7cbdff675d29"} Nov 01 12:59:48.649590 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.649590 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84f0da22bf302129075ac121d6ac09e63fecfd124c159e087ce7cbdff675d29" Nov 01 12:59:48.650938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.650911 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" event={"ID":"6a624144-2e97-47c7-b648-3129e4d1c821","Type":"ContainerDied","Data":"0220b08ffb992cd2c56ea7708ed89043ea70497a843ab9fd5a61b3b8d726221c"} Nov 01 12:59:48.650938 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.650933 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8" Nov 01 12:59:48.651133 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:48.650953 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0220b08ffb992cd2c56ea7708ed89043ea70497a843ab9fd5a61b3b8d726221c" Nov 01 12:59:51.403865 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:51.403835 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:51.944783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:51.944759 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:59:52.121510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.121483 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds\" (UniqueName: \"kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds\") pod \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " Nov 01 12:59:52.121701 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.121604 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume\") pod \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\" (UID: \"c09fa82a-c9e6-4f24-915c-f17b3f3cc969\") " Nov 01 12:59:52.122001 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.121973 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume" pod "c09fa82a-c9e6-4f24-915c-f17b3f3cc969" (UID: "c09fa82a-c9e6-4f24-915c-f17b3f3cc969"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:52.123509 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.123484 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds" pod "c09fa82a-c9e6-4f24-915c-f17b3f3cc969" (UID: "c09fa82a-c9e6-4f24-915c-f17b3f3cc969"). InnerVolumeSpecName "kube-api-access-mbgds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:52.222874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.222807 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:52.222874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.222835 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbgds\" (UniqueName: \"kubernetes.io/projected/c09fa82a-c9e6-4f24-915c-f17b3f3cc969-kube-api-access-mbgds\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:52.417395 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.417362 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:52.537288 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.537258 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:52.680934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.680893 4484 generic.go:356] "Generic (PLEG): container finished" podID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerID="e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4" exitCode=0 Nov 01 12:59:52.680934 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.680937 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" Nov 01 12:59:52.681189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.680930 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" event={"ID":"c09fa82a-c9e6-4f24-915c-f17b3f3cc969","Type":"ContainerDied","Data":"e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4"} Nov 01 12:59:52.681189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.681076 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr" event={"ID":"c09fa82a-c9e6-4f24-915c-f17b3f3cc969","Type":"ContainerDied","Data":"8bf984804636218a01c42c2d811d723a247b53c5e024ef30e7c39167ce1f820a"} Nov 01 12:59:52.681189 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.681108 4484 scope.go:122] "RemoveContainer" containerID="e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4" Nov 01 12:59:52.690847 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.690828 4484 scope.go:122] "RemoveContainer" containerID="e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4" Nov 01 12:59:52.691120 ip-10-0-140-174 kubenswrapper[4484]: E1101 12:59:52.691098 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4\": container with ID starting with e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4 not found: ID does not exist" containerID="e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4" Nov 01 12:59:52.691163 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.691127 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4"} err="failed to get container status \"e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4\": rpc error: code = NotFound desc = could not find container \"e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4\": container with ID starting with e935ac611a33982ebae2857b3a9d3b930cb21b5de61e70e35abb29d9d191f3d4 not found: ID does not exist" Nov 01 12:59:52.833986 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.833905 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr"] Nov 01 12:59:52.880039 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:52.880012 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-94c94b-7787ccf945-dbfrr"] Nov 01 12:59:53.760878 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:53.760847 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:53.898102 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:53.898072 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:54.022076 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:54.021999 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:54.142951 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:54.142920 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:54.266815 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:54.266783 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:54.368716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:54.368645 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" path="/var/lib/kubelet/pods/c09fa82a-c9e6-4f24-915c-f17b3f3cc969/volumes" Nov 01 12:59:54.529997 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:54.529967 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:54.650572 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:54.650489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:55.122220 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122186 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87"] Nov 01 12:59:55.122783 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122767 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerName="payload-processing" Nov 01 12:59:55.122837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122784 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerName="payload-processing" Nov 01 12:59:55.122837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122800 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6a624144-2e97-47c7-b648-3129e4d1c821" containerName="revoke-keys" Nov 01 12:59:55.122837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122805 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a624144-2e97-47c7-b648-3129e4d1c821" containerName="revoke-keys" Nov 01 12:59:55.122837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122830 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b0a3201b-a468-41ae-b865-8913bf516c11" containerName="revoke-keys" Nov 01 12:59:55.122837 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122835 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a3201b-a468-41ae-b865-8913bf516c11" containerName="revoke-keys" Nov 01 12:59:55.122993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122906 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09fa82a-c9e6-4f24-915c-f17b3f3cc969" containerName="payload-processing" Nov 01 12:59:55.122993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122917 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a624144-2e97-47c7-b648-3129e4d1c821" containerName="revoke-keys" Nov 01 12:59:55.122993 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.122928 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a3201b-a468-41ae-b865-8913bf516c11" containerName="revoke-keys" Nov 01 12:59:55.127487 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.127467 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.162646 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.162619 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgm4\" (UniqueName: \"kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.162741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.162658 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.162741 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.162686 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.192514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.192467 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 12:59:55.208115 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.207947 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87"] Nov 01 12:59:55.236018 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.235986 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 12:59:55.264036 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.264002 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgm4\" (UniqueName: \"kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.264200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.264053 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.264200 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.264082 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.264943 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.264912 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.267957 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.267910 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.320566 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.320539 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgm4\" (UniqueName: \"kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.440992 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.440926 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:55.482547 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.482475 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf"] Nov 01 12:59:55.487984 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.487962 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.541973 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.541941 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b68322-openshift-default-dockercfg-fjb4c\"" Nov 01 12:59:55.551631 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.551603 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf"] Nov 01 12:59:55.574357 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.574243 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:55.668032 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.667986 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668048 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668117 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxx98\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668186 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668230 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668210 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668238 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668283 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668377 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.668514 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.668467 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770156 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770128 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770167 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770196 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770240 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx98\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770294 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770349 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770333 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770357 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770405 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770511 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770551 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770660 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770578 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770694 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.770915 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.770869 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.771118 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.771099 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.772570 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.772552 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.772944 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.772918 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.817833 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.817810 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87"] Nov 01 12:59:55.823773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.823754 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.823773 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.823766 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx98\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98\") pod \"e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:55.824704 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:55.824679 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7440900_b982_4cba_b878_46f3baf28baf.slice/crio-c8b0decbfbd400eb1c39a670e02ad9a164330bb5b9beac0f957d26d918bb7200 WatchSource:0}: Error finding container c8b0decbfbd400eb1c39a670e02ad9a164330bb5b9beac0f957d26d918bb7200: Status 404 returned error can't find the container with id c8b0decbfbd400eb1c39a670e02ad9a164330bb5b9beac0f957d26d918bb7200 Nov 01 12:59:55.832683 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:55.832659 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:56.106060 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.106032 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:56.534925 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.534886 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf"] Nov 01 12:59:56.549467 ip-10-0-140-174 kubenswrapper[4484]: W1101 12:59:56.544105 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8443af5_fdb8_4072_9e1b_567609af5e6e.slice/crio-08af3cd85253183d55ad135d760348ffc7c0de0288a61416119e3f6c2604cd9a WatchSource:0}: Error finding container 08af3cd85253183d55ad135d760348ffc7c0de0288a61416119e3f6c2604cd9a: Status 404 returned error can't find the container with id 08af3cd85253183d55ad135d760348ffc7c0de0288a61416119e3f6c2604cd9a Nov 01 12:59:56.549467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.547664 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:59:56.549467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.547749 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:59:56.549467 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.547790 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 12:59:56.709849 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.709806 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" event={"ID":"e8443af5-fdb8-4072-9e1b-567609af5e6e","Type":"ContainerStarted","Data":"eb6b4c103648e7bd4ccecc67e8340243ef2510d321cafc135a1d41250c6e6e29"} Nov 01 12:59:56.710044 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.709907 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" event={"ID":"e8443af5-fdb8-4072-9e1b-567609af5e6e","Type":"ContainerStarted","Data":"08af3cd85253183d55ad135d760348ffc7c0de0288a61416119e3f6c2604cd9a"} Nov 01 12:59:56.711874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.711743 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" event={"ID":"d7440900-b982-4cba-b878-46f3baf28baf","Type":"ContainerStarted","Data":"88df0620e24ca8a8d1f1615ff7a17e3c201751f44b5ba525112df219880e7147"} Nov 01 12:59:56.711874 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.711797 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" event={"ID":"d7440900-b982-4cba-b878-46f3baf28baf","Type":"ContainerStarted","Data":"c8b0decbfbd400eb1c39a670e02ad9a164330bb5b9beac0f957d26d918bb7200"} Nov 01 12:59:56.824371 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.824245 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podStartSLOduration=1.824226898 podStartE2EDuration="1.824226898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:56.819997237 +0000 UTC m=+2266.569940216" watchObservedRunningTime="2026-11-01 12:59:56.824226898 +0000 UTC m=+2266.574169876" Nov 01 12:59:56.932464 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:56.932406 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" podStartSLOduration=1.932392106 podStartE2EDuration="1.932392106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:56.930614411 +0000 UTC m=+2266.680557389" watchObservedRunningTime="2026-11-01 12:59:56.932392106 +0000 UTC m=+2266.682335084" Nov 01 12:59:57.107177 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.107099 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:57.112452 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.112428 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:57.120825 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.120803 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:57.306432 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.306400 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:57.421716 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.421626 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:57.719321 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.719237 4484 generic.go:356] "Generic (PLEG): container finished" podID="d7440900-b982-4cba-b878-46f3baf28baf" containerID="88df0620e24ca8a8d1f1615ff7a17e3c201751f44b5ba525112df219880e7147" exitCode=0 Nov 01 12:59:57.719685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.719339 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" event={"ID":"d7440900-b982-4cba-b878-46f3baf28baf","Type":"ContainerDied","Data":"88df0620e24ca8a8d1f1615ff7a17e3c201751f44b5ba525112df219880e7147"} Nov 01 12:59:57.719685 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.719634 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:57.720589 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:57.720568 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 12:59:58.874820 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.874789 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:58.909046 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.909022 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4\" (UniqueName: \"kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4\") pod \"d7440900-b982-4cba-b878-46f3baf28baf\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " Nov 01 12:59:58.909163 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.909061 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca\") pod \"d7440900-b982-4cba-b878-46f3baf28baf\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " Nov 01 12:59:58.909163 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.909127 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca\") pod \"d7440900-b982-4cba-b878-46f3baf28baf\" (UID: \"d7440900-b982-4cba-b878-46f3baf28baf\") " Nov 01 12:59:58.909466 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.909440 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca" pod "d7440900-b982-4cba-b878-46f3baf28baf" (UID: "d7440900-b982-4cba-b878-46f3baf28baf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 12:59:58.909698 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.909672 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:58.911069 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.911048 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4" pod "d7440900-b982-4cba-b878-46f3baf28baf" (UID: "d7440900-b982-4cba-b878-46f3baf28baf"). InnerVolumeSpecName "kube-api-access-2kgm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 12:59:58.911157 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:58.911106 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca" pod "d7440900-b982-4cba-b878-46f3baf28baf" (UID: "d7440900-b982-4cba-b878-46f3baf28baf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 12:59:59.010551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.010491 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7440900-b982-4cba-b878-46f3baf28baf-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:59.010551 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.010516 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kgm4\" (UniqueName: \"kubernetes.io/projected/d7440900-b982-4cba-b878-46f3baf28baf-kube-api-access-2kgm4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 12:59:59.521062 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.521016 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj"] Nov 01 12:59:59.597537 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.597487 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:59.715517 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.715469 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 12:59:59.734510 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.734482 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" Nov 01 12:59:59.734643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.734513 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87" event={"ID":"d7440900-b982-4cba-b878-46f3baf28baf","Type":"ContainerDied","Data":"c8b0decbfbd400eb1c39a670e02ad9a164330bb5b9beac0f957d26d918bb7200"} Nov 01 12:59:59.734643 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.734561 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b0decbfbd400eb1c39a670e02ad9a164330bb5b9beac0f957d26d918bb7200" Nov 01 12:59:59.865614 ip-10-0-140-174 kubenswrapper[4484]: I1101 12:59:59.865526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:00.010883 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.010853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:00.162454 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.162373 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9"] Nov 01 13:00:00.162900 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.162887 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d7440900-b982-4cba-b878-46f3baf28baf" containerName="revoke-keys" Nov 01 13:00:00.162940 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.162901 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7440900-b982-4cba-b878-46f3baf28baf" containerName="revoke-keys" Nov 01 13:00:00.163005 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.162996 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7440900-b982-4cba-b878-46f3baf28baf" containerName="revoke-keys" Nov 01 13:00:00.166459 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.166440 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.168917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.168456 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2"] Nov 01 13:00:00.174409 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.174388 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.175259 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.175237 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:00.179181 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.179054 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp"] Nov 01 13:00:00.184867 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.184846 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf"] Nov 01 13:00:00.185059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.185039 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.188824 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.188806 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.203735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.203714 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md"] Nov 01 13:00:00.208145 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.208124 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn"] Nov 01 13:00:00.208251 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.208227 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.212648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.212627 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.221258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.221226 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:00:00.222802 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.222778 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9"] Nov 01 13:00:00.238659 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.231781 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2"] Nov 01 13:00:00.253711 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.253674 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.253905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.253748 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.253905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.253856 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.254041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.253987 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpmp\" (UniqueName: \"kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.254041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.254033 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fnl\" (UniqueName: \"kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.254149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.254061 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgxx\" (UniqueName: \"kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx\") pod \"maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.254149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.254112 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.254279 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.254258 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.254390 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.254371 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6455\" (UniqueName: \"kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455\") pod \"maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.254448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.254404 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kn6n\" (UniqueName: \"kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.259495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.259469 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp"] Nov 01 13:00:00.263225 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.263200 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf"] Nov 01 13:00:00.266627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.266604 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md"] Nov 01 13:00:00.269509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.269488 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn"] Nov 01 13:00:00.291781 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:00.291746 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:00:00.308807 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.308778 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:00.355923 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.355888 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.356064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.355968 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6455\" (UniqueName: \"kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455\") pod \"maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.356064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356001 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn6n\" (UniqueName: \"kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.356064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356038 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356072 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjx4j\" (UniqueName: \"kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356116 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356144 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356186 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpmp\" (UniqueName: \"kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356227 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fnl\" (UniqueName: \"kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356255 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgxx\" (UniqueName: \"kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx\") pod \"maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356321 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.356364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356347 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.356796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356771 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.356876 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356855 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.356974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.356955 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.357098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.357079 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.357438 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.357418 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.412850 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.412783 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn6n\" (UniqueName: \"kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.413024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.412869 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6455\" (UniqueName: \"kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455\") pod \"maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.413024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.412935 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fnl\" (UniqueName: \"kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.413160 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.413088 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgxx\" (UniqueName: \"kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx\") pod \"maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.414401 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.414379 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpmp\" (UniqueName: \"kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp\") pod \"maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.457800 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.457770 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.457926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.457910 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjx4j\" (UniqueName: \"kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.458477 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.458460 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.487609 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.487583 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:00.498926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.498904 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:00.506223 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.506199 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:00.511442 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.511416 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjx4j\" (UniqueName: \"kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.514171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.514147 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:00.527020 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.526863 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:00.557780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.557728 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:00.879282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:00.879240 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9"] Nov 01 13:00:00.883998 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:00.883964 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879f8d85_3a00_4665_8790_59fd23cffd53.slice/crio-43df97d0bdab03cb931296183b4539ecc96f49f51d3e6a3bdcce07d7ab46f576 WatchSource:0}: Error finding container 43df97d0bdab03cb931296183b4539ecc96f49f51d3e6a3bdcce07d7ab46f576: Status 404 returned error can't find the container with id 43df97d0bdab03cb931296183b4539ecc96f49f51d3e6a3bdcce07d7ab46f576 Nov 01 13:00:01.014841 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.014796 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp"] Nov 01 13:00:01.018142 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.018117 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf"] Nov 01 13:00:01.020384 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:01.020345 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5cad83_6f70_40bb_ba6b_e7da46fe3a3a.slice/crio-820fde903e53d3028727e8d6e0e9d53c15782773ac405519de00a7beedbdfab6 WatchSource:0}: Error finding container 820fde903e53d3028727e8d6e0e9d53c15782773ac405519de00a7beedbdfab6: Status 404 returned error can't find the container with id 820fde903e53d3028727e8d6e0e9d53c15782773ac405519de00a7beedbdfab6 Nov 01 13:00:01.022971 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:01.022940 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a4ab08_5544_4d77_bcb6_efe88470f9ec.slice/crio-c47f4eeb17fe48e546115c0fe5cc07dbffb395bab02fff68c93a540938e148f9 WatchSource:0}: Error finding container c47f4eeb17fe48e546115c0fe5cc07dbffb395bab02fff68c93a540938e148f9: Status 404 returned error can't find the container with id c47f4eeb17fe48e546115c0fe5cc07dbffb395bab02fff68c93a540938e148f9 Nov 01 13:00:01.067704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.067681 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md"] Nov 01 13:00:01.073136 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:01.073113 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6493c38_f301_4c38_879d_6e7459179855.slice/crio-b69be058bce5e9f7a44d854012c916a321f7b31f8689926eab90902528bf09d0 WatchSource:0}: Error finding container b69be058bce5e9f7a44d854012c916a321f7b31f8689926eab90902528bf09d0: Status 404 returned error can't find the container with id b69be058bce5e9f7a44d854012c916a321f7b31f8689926eab90902528bf09d0 Nov 01 13:00:01.080177 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.080151 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn"] Nov 01 13:00:01.086092 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:01.086065 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2fa7b7_8178_458c_b62b_e19675c4b2bf.slice/crio-634c54d81e6b5c859c908cfeecfc5894480ac2593869e52d961a09108ac9b331 WatchSource:0}: Error finding container 634c54d81e6b5c859c908cfeecfc5894480ac2593869e52d961a09108ac9b331: Status 404 returned error can't find the container with id 634c54d81e6b5c859c908cfeecfc5894480ac2593869e52d961a09108ac9b331 Nov 01 13:00:01.121027 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.121005 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2"] Nov 01 13:00:01.126582 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:01.126558 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabbbce6_c46c_459f_99ef_6fce9ca2f4b0.slice/crio-1054081a10781d5d56a0f7c1a7e0eea6c3c4175221192684fbd43311d38c830c WatchSource:0}: Error finding container 1054081a10781d5d56a0f7c1a7e0eea6c3c4175221192684fbd43311d38c830c: Status 404 returned error can't find the container with id 1054081a10781d5d56a0f7c1a7e0eea6c3c4175221192684fbd43311d38c830c Nov 01 13:00:01.762462 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.762421 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerStarted","Data":"c72b5ea277c13ab867ae97b93896706be61eb236b093b731c5940e21e916ac1f"} Nov 01 13:00:01.762584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.762495 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerStarted","Data":"634c54d81e6b5c859c908cfeecfc5894480ac2593869e52d961a09108ac9b331"} Nov 01 13:00:01.764002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.763975 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerStarted","Data":"430e6ac5165df5a4d6d1598b8beb58ba561fa111e316484e88a12d1d420b192d"} Nov 01 13:00:01.764125 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.764015 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerStarted","Data":"c47f4eeb17fe48e546115c0fe5cc07dbffb395bab02fff68c93a540938e148f9"} Nov 01 13:00:01.765434 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.765407 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" event={"ID":"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0","Type":"ContainerStarted","Data":"a086c02c4857eb18aac6346801861919ad06d18475cdde80772fb5c0b9dad3f4"} Nov 01 13:00:01.765539 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.765452 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" event={"ID":"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0","Type":"ContainerStarted","Data":"1054081a10781d5d56a0f7c1a7e0eea6c3c4175221192684fbd43311d38c830c"} Nov 01 13:00:01.770536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.770512 4484 generic.go:356] "Generic (PLEG): container finished" podID="879f8d85-3a00-4665-8790-59fd23cffd53" containerID="62e2acf594a3f1c750e7646fb31686941f04ab5291f670c8965c5c5f64152302" exitCode=60 Nov 01 13:00:01.770656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.770590 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerDied","Data":"62e2acf594a3f1c750e7646fb31686941f04ab5291f670c8965c5c5f64152302"} Nov 01 13:00:01.770656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.770628 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerStarted","Data":"43df97d0bdab03cb931296183b4539ecc96f49f51d3e6a3bdcce07d7ab46f576"} Nov 01 13:00:01.770813 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.770795 4484 scope.go:122] "RemoveContainer" containerID="62e2acf594a3f1c750e7646fb31686941f04ab5291f670c8965c5c5f64152302" Nov 01 13:00:01.772309 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.772276 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerStarted","Data":"f57b0e7e46f73195866a31669f966e39847b003c8114285d4ef38c7accacbfde"} Nov 01 13:00:01.772386 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.772348 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerStarted","Data":"b69be058bce5e9f7a44d854012c916a321f7b31f8689926eab90902528bf09d0"} Nov 01 13:00:01.783681 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.783650 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerStarted","Data":"187766d36dddd56c4e339c1b817c6dcd972f448927bbd7b6db5d4d38e0050354"} Nov 01 13:00:01.783789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.783729 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerStarted","Data":"820fde903e53d3028727e8d6e0e9d53c15782773ac405519de00a7beedbdfab6"} Nov 01 13:00:01.899950 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.899894 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" podStartSLOduration=1.899879866 podStartE2EDuration="1.899879866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:01.890291191 +0000 UTC m=+2271.640234168" watchObservedRunningTime="2026-11-01 13:00:01.899879866 +0000 UTC m=+2271.649822844" Nov 01 13:00:01.999490 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:01.999393 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" podStartSLOduration=1.9993788860000001 podStartE2EDuration="1.999378886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:01.996979635 +0000 UTC m=+2271.746922615" watchObservedRunningTime="2026-11-01 13:00:01.999378886 +0000 UTC m=+2271.749321863" Nov 01 13:00:02.106840 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.106794 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" podStartSLOduration=2.106781174 podStartE2EDuration="2.106781174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.104514406 +0000 UTC m=+2271.854457384" watchObservedRunningTime="2026-11-01 13:00:02.106781174 +0000 UTC m=+2271.856724153" Nov 01 13:00:02.210741 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.210688 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" podStartSLOduration=2.210675305 podStartE2EDuration="2.210675305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.210014136 +0000 UTC m=+2271.959957114" watchObservedRunningTime="2026-11-01 13:00:02.210675305 +0000 UTC m=+2271.960618283" Nov 01 13:00:02.320710 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.320662 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" podStartSLOduration=2.320647145 podStartE2EDuration="2.320647145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.317611963 +0000 UTC m=+2272.067554942" watchObservedRunningTime="2026-11-01 13:00:02.320647145 +0000 UTC m=+2272.070590123" Nov 01 13:00:02.538455 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.538355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:02.793104 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.793020 4484 generic.go:356] "Generic (PLEG): container finished" podID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerID="c72b5ea277c13ab867ae97b93896706be61eb236b093b731c5940e21e916ac1f" exitCode=60 Nov 01 13:00:02.793259 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.793096 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerDied","Data":"c72b5ea277c13ab867ae97b93896706be61eb236b093b731c5940e21e916ac1f"} Nov 01 13:00:02.793377 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.793361 4484 scope.go:122] "RemoveContainer" containerID="c72b5ea277c13ab867ae97b93896706be61eb236b093b731c5940e21e916ac1f" Nov 01 13:00:02.794706 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.794687 4484 generic.go:356] "Generic (PLEG): container finished" podID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerID="430e6ac5165df5a4d6d1598b8beb58ba561fa111e316484e88a12d1d420b192d" exitCode=60 Nov 01 13:00:02.794804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.794775 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerDied","Data":"430e6ac5165df5a4d6d1598b8beb58ba561fa111e316484e88a12d1d420b192d"} Nov 01 13:00:02.795001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.794985 4484 scope.go:122] "RemoveContainer" containerID="430e6ac5165df5a4d6d1598b8beb58ba561fa111e316484e88a12d1d420b192d" Nov 01 13:00:02.796490 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.796472 4484 generic.go:356] "Generic (PLEG): container finished" podID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerID="a086c02c4857eb18aac6346801861919ad06d18475cdde80772fb5c0b9dad3f4" exitCode=60 Nov 01 13:00:02.796569 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.796539 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" event={"ID":"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0","Type":"ContainerDied","Data":"a086c02c4857eb18aac6346801861919ad06d18475cdde80772fb5c0b9dad3f4"} Nov 01 13:00:02.796724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.796688 4484 scope.go:122] "RemoveContainer" containerID="a086c02c4857eb18aac6346801861919ad06d18475cdde80772fb5c0b9dad3f4" Nov 01 13:00:02.799124 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.799104 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerStarted","Data":"d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834"} Nov 01 13:00:02.800792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.800773 4484 generic.go:356] "Generic (PLEG): container finished" podID="a6493c38-f301-4c38-879d-6e7459179855" containerID="f57b0e7e46f73195866a31669f966e39847b003c8114285d4ef38c7accacbfde" exitCode=60 Nov 01 13:00:02.800891 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.800833 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerDied","Data":"f57b0e7e46f73195866a31669f966e39847b003c8114285d4ef38c7accacbfde"} Nov 01 13:00:02.801078 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.801062 4484 scope.go:122] "RemoveContainer" containerID="f57b0e7e46f73195866a31669f966e39847b003c8114285d4ef38c7accacbfde" Nov 01 13:00:02.802589 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.802566 4484 generic.go:356] "Generic (PLEG): container finished" podID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerID="187766d36dddd56c4e339c1b817c6dcd972f448927bbd7b6db5d4d38e0050354" exitCode=7 Nov 01 13:00:02.802689 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.802643 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerDied","Data":"187766d36dddd56c4e339c1b817c6dcd972f448927bbd7b6db5d4d38e0050354"} Nov 01 13:00:02.802848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.802832 4484 scope.go:122] "RemoveContainer" containerID="187766d36dddd56c4e339c1b817c6dcd972f448927bbd7b6db5d4d38e0050354" Nov 01 13:00:02.975857 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:02.975825 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:03.169419 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.161425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:03.351758 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.351636 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" podStartSLOduration=3.35162099 podStartE2EDuration="3.35162099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:03.351161691 +0000 UTC m=+2273.101104662" watchObservedRunningTime="2026-11-01 13:00:03.35162099 +0000 UTC m=+2273.101563968" Nov 01 13:00:03.412419 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.412386 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw"] Nov 01 13:00:03.416592 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.416571 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.468015 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.467986 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b68322-dockercfg-mq7tj\"" Nov 01 13:00:03.468015 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.468004 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b68322\"" Nov 01 13:00:03.472060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.472036 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw"] Nov 01 13:00:03.532451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.532378 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwtx\" (UniqueName: \"kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx\") pod \"payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.532762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.532729 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.572176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.572122 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k"] Nov 01 13:00:03.586971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.586941 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.612446 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.612384 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k"] Nov 01 13:00:03.634083 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.634047 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwtx\" (UniqueName: \"kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx\") pod \"payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.634216 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.634174 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.635269 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.635247 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.687342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.687286 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwtx\" (UniqueName: \"kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx\") pod \"payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.735197 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.735169 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxk5f\" (UniqueName: \"kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f\") pod \"payload-processing-e2e-inf-b-b68322-794645fb85-4h74k\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.735197 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.735183 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:03.735451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.735232 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b68322-794645fb85-4h74k\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.820696 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.820666 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerStarted","Data":"8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5"} Nov 01 13:00:03.822615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.822584 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerStarted","Data":"29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479"} Nov 01 13:00:03.824718 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.824683 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" event={"ID":"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0","Type":"ContainerStarted","Data":"2846c58edc1ed355fa3d0f5e4b50abc6f5dcd65cf3004d48d6e438a4e179c3c5"} Nov 01 13:00:03.826954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.826924 4484 generic.go:356] "Generic (PLEG): container finished" podID="879f8d85-3a00-4665-8790-59fd23cffd53" containerID="d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834" exitCode=60 Nov 01 13:00:03.827085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.827057 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerDied","Data":"d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834"} Nov 01 13:00:03.827194 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.827105 4484 scope.go:122] "RemoveContainer" containerID="62e2acf594a3f1c750e7646fb31686941f04ab5291f670c8965c5c5f64152302" Nov 01 13:00:03.827273 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.827239 4484 scope.go:122] "RemoveContainer" containerID="d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834" Nov 01 13:00:03.827578 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:03.827552 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9_odh-ai-gateway-infra(879f8d85-3a00-4665-8790-59fd23cffd53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" Nov 01 13:00:03.829365 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.829325 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:03.829469 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.829428 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerStarted","Data":"181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129"} Nov 01 13:00:03.832711 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.832690 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerStarted","Data":"6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15"} Nov 01 13:00:03.836542 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.836513 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxk5f\" (UniqueName: \"kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f\") pod \"payload-processing-e2e-inf-b-b68322-794645fb85-4h74k\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.836637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.836588 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b68322-794645fb85-4h74k\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.837223 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.837202 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b68322-794645fb85-4h74k\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.889531 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.889507 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxk5f\" (UniqueName: \"kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f\") pod \"payload-processing-e2e-inf-b-b68322-794645fb85-4h74k\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.904530 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.904501 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:03.931157 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:03.931131 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2"] Nov 01 13:00:04.186896 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.186561 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw"] Nov 01 13:00:04.305910 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.305875 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k"] Nov 01 13:00:04.311619 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:04.311592 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86080597_c9d7_40b3_975e_31ed344ba8d1.slice/crio-b4a2a939c16465bd045bbaaf11b5f945a6440a3b757cfa7095037e498767522a WatchSource:0}: Error finding container b4a2a939c16465bd045bbaaf11b5f945a6440a3b757cfa7095037e498767522a: Status 404 returned error can't find the container with id b4a2a939c16465bd045bbaaf11b5f945a6440a3b757cfa7095037e498767522a Nov 01 13:00:04.665587 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.665553 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:04.841862 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.841831 4484 generic.go:356] "Generic (PLEG): container finished" podID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerID="8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5" exitCode=60 Nov 01 13:00:04.842057 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.841912 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerDied","Data":"8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5"} Nov 01 13:00:04.842057 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.841987 4484 scope.go:122] "RemoveContainer" containerID="c72b5ea277c13ab867ae97b93896706be61eb236b093b731c5940e21e916ac1f" Nov 01 13:00:04.842344 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.842294 4484 scope.go:122] "RemoveContainer" containerID="8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5" Nov 01 13:00:04.842631 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:04.842607 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn_odh-ai-gateway-infra(6c2fa7b7-8178-458c-b62b-e19675c4b2bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" Nov 01 13:00:04.844563 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.844543 4484 generic.go:356] "Generic (PLEG): container finished" podID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerID="29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479" exitCode=60 Nov 01 13:00:04.844808 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.844789 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerDied","Data":"29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479"} Nov 01 13:00:04.844993 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.844974 4484 scope.go:122] "RemoveContainer" containerID="29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479" Nov 01 13:00:04.845359 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:04.845296 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf_odh-ai-gateway-infra(b1a4ab08-5544-4d77-bcb6-efe88470f9ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" Nov 01 13:00:04.848158 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.848136 4484 generic.go:356] "Generic (PLEG): container finished" podID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerID="2846c58edc1ed355fa3d0f5e4b50abc6f5dcd65cf3004d48d6e438a4e179c3c5" exitCode=7 Nov 01 13:00:04.848280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.848210 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" event={"ID":"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0","Type":"ContainerDied","Data":"2846c58edc1ed355fa3d0f5e4b50abc6f5dcd65cf3004d48d6e438a4e179c3c5"} Nov 01 13:00:04.848466 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.848451 4484 scope.go:122] "RemoveContainer" containerID="2846c58edc1ed355fa3d0f5e4b50abc6f5dcd65cf3004d48d6e438a4e179c3c5" Nov 01 13:00:04.848681 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:04.848663 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2_odh-ai-gateway-infra(dabbbce6-c46c-459f-99ef-6fce9ca2f4b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" Nov 01 13:00:04.850671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.850652 4484 scope.go:122] "RemoveContainer" containerID="d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834" Nov 01 13:00:04.851022 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:04.851001 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9_odh-ai-gateway-infra(879f8d85-3a00-4665-8790-59fd23cffd53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" Nov 01 13:00:04.854205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.854174 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" event={"ID":"86080597-c9d7-40b3-975e-31ed344ba8d1","Type":"ContainerStarted","Data":"8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1"} Nov 01 13:00:04.854360 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.854231 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" event={"ID":"86080597-c9d7-40b3-975e-31ed344ba8d1","Type":"ContainerStarted","Data":"b4a2a939c16465bd045bbaaf11b5f945a6440a3b757cfa7095037e498767522a"} Nov 01 13:00:04.854888 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.854864 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:04.857253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.857235 4484 generic.go:356] "Generic (PLEG): container finished" podID="a6493c38-f301-4c38-879d-6e7459179855" containerID="181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129" exitCode=60 Nov 01 13:00:04.857382 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.857329 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerDied","Data":"181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129"} Nov 01 13:00:04.857561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.857548 4484 scope.go:122] "RemoveContainer" containerID="181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129" Nov 01 13:00:04.857797 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:04.857779 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md_odh-ai-gateway-infra(a6493c38-f301-4c38-879d-6e7459179855)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" podUID="a6493c38-f301-4c38-879d-6e7459179855" Nov 01 13:00:04.859387 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.859362 4484 generic.go:356] "Generic (PLEG): container finished" podID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerID="6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15" exitCode=7 Nov 01 13:00:04.859545 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.859390 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerDied","Data":"6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15"} Nov 01 13:00:04.860021 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.860004 4484 scope.go:122] "RemoveContainer" containerID="6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15" Nov 01 13:00:04.860280 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:04.860260 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp_odh-ai-gateway-infra(bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" Nov 01 13:00:04.860450 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.860430 4484 scope.go:122] "RemoveContainer" containerID="430e6ac5165df5a4d6d1598b8beb58ba561fa111e316484e88a12d1d420b192d" Nov 01 13:00:04.861716 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.861697 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" event={"ID":"1419fb27-10bb-4263-8ac5-6f2663975eb3","Type":"ContainerStarted","Data":"ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881"} Nov 01 13:00:04.861798 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.861731 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" event={"ID":"1419fb27-10bb-4263-8ac5-6f2663975eb3","Type":"ContainerStarted","Data":"272bfef82c01bb08326515313d8c8cd755ad39cd5003ae5d9885a2e668ed221e"} Nov 01 13:00:04.861798 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.861790 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:04.875266 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.875250 4484 scope.go:122] "RemoveContainer" containerID="a086c02c4857eb18aac6346801861919ad06d18475cdde80772fb5c0b9dad3f4" Nov 01 13:00:04.887803 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.887785 4484 scope.go:122] "RemoveContainer" containerID="f57b0e7e46f73195866a31669f966e39847b003c8114285d4ef38c7accacbfde" Nov 01 13:00:04.899891 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.899876 4484 scope.go:122] "RemoveContainer" containerID="187766d36dddd56c4e339c1b817c6dcd972f448927bbd7b6db5d4d38e0050354" Nov 01 13:00:04.956545 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:04.956494 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:05.219617 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.219526 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg"] Nov 01 13:00:05.227727 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.227689 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78c69d-wnpxg"] Nov 01 13:00:05.269345 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.269269 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" podStartSLOduration=2.269255942 podStartE2EDuration="2.269255942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:05.268506187 +0000 UTC m=+2275.018449164" watchObservedRunningTime="2026-11-01 13:00:05.269255942 +0000 UTC m=+2275.019198930" Nov 01 13:00:05.602637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.602607 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:05.703185 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.703136 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" podStartSLOduration=2.703117545 podStartE2EDuration="2.703117545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:05.701905016 +0000 UTC m=+2275.451847995" watchObservedRunningTime="2026-11-01 13:00:05.703117545 +0000 UTC m=+2275.453060523" Nov 01 13:00:05.788230 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.788197 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:05.871405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.871294 4484 scope.go:122] "RemoveContainer" containerID="181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129" Nov 01 13:00:05.871651 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:05.871623 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md_odh-ai-gateway-infra(a6493c38-f301-4c38-879d-6e7459179855)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" podUID="a6493c38-f301-4c38-879d-6e7459179855" Nov 01 13:00:05.873141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.873120 4484 scope.go:122] "RemoveContainer" containerID="6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15" Nov 01 13:00:05.873375 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:05.873358 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp_odh-ai-gateway-infra(bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" Nov 01 13:00:05.875648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.875629 4484 scope.go:122] "RemoveContainer" containerID="8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5" Nov 01 13:00:05.875831 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:05.875812 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn_odh-ai-gateway-infra(6c2fa7b7-8178-458c-b62b-e19675c4b2bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" Nov 01 13:00:05.877204 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.877189 4484 scope.go:122] "RemoveContainer" containerID="29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479" Nov 01 13:00:05.877425 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:05.877404 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf_odh-ai-gateway-infra(b1a4ab08-5544-4d77-bcb6-efe88470f9ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" Nov 01 13:00:05.923385 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:05.923347 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:06.055752 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.055726 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:06.130388 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.130292 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:06.165597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.165561 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca\") pod \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " Nov 01 13:00:06.165754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.165655 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455\" (UniqueName: \"kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455\") pod \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\" (UID: \"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0\") " Nov 01 13:00:06.165900 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.165879 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca" pod "dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" (UID: "dabbbce6-c46c-459f-99ef-6fce9ca2f4b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:06.166089 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.166071 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:06.167666 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.167641 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455" pod "dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" (UID: "dabbbce6-c46c-459f-99ef-6fce9ca2f4b0"). InnerVolumeSpecName "kube-api-access-z6455". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:06.266890 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.266858 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6455\" (UniqueName: \"kubernetes.io/projected/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0-kube-api-access-z6455\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:06.371127 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.371096 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33b08c3-1d24-4659-a46f-70587947cd17" path="/var/lib/kubelet/pods/d33b08c3-1d24-4659-a46f-70587947cd17/volumes" Nov 01 13:00:06.711849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.711819 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:06.886162 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.886130 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" Nov 01 13:00:06.886341 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.886176 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2" event={"ID":"dabbbce6-c46c-459f-99ef-6fce9ca2f4b0","Type":"ContainerDied","Data":"1054081a10781d5d56a0f7c1a7e0eea6c3c4175221192684fbd43311d38c830c"} Nov 01 13:00:06.886341 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:06.886235 4484 scope.go:122] "RemoveContainer" containerID="2846c58edc1ed355fa3d0f5e4b50abc6f5dcd65cf3004d48d6e438a4e179c3c5" Nov 01 13:00:07.038931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:07.038904 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2"] Nov 01 13:00:07.042034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:07.042012 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-94c13a-29892300-ns9j2"] Nov 01 13:00:08.370325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:08.370274 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" path="/var/lib/kubelet/pods/dabbbce6-c46c-459f-99ef-6fce9ca2f4b0/volumes" Nov 01 13:00:11.051016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:11.050981 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz"] Nov 01 13:00:11.159143 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:11.159113 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj"] Nov 01 13:00:11.398653 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:11.398581 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:11.513208 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:11.513173 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:11.883295 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:11.883266 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:12.151571 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:12.151478 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:12.248521 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:12.248486 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:00:12.284186 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:12.284137 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:12.500766 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:12.500285 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:12.874891 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:12.874797 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:13.362816 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:13.362782 4484 scope.go:122] "RemoveContainer" containerID="d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834" Nov 01 13:00:13.387194 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:13.387170 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:13.509167 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:13.509134 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:13.639488 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:13.639409 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:13.938770 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:13.938695 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerStarted","Data":"29f700565167670445639e22911c9aa562ab7d658b418a21a5e70ab826b24030"} Nov 01 13:00:14.408834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.408804 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:14.542085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.542043 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" containerID="cri-o://443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c" gracePeriod=45 Nov 01 13:00:14.543769 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.543742 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:14.768342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.765775 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc"] Nov 01 13:00:14.768342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.766005 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" containerID="cri-o://7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813" gracePeriod=30 Nov 01 13:00:14.778198 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.777044 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:14.832816 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.832791 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941129 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941203 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir\") pod \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941263 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home\") pod \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941321 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m\" (UniqueName: \"kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m\") pod \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941424 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache\") pod \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941449 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs\") pod \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941468 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm\") pod \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\" (UID: \"aa169038-e9bd-47c5-b6dc-093f8a1d9c08\") " Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941596 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir" pod "aa169038-e9bd-47c5-b6dc-093f8a1d9c08" (UID: "aa169038-e9bd-47c5-b6dc-093f8a1d9c08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941808 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache" pod "aa169038-e9bd-47c5-b6dc-093f8a1d9c08" (UID: "aa169038-e9bd-47c5-b6dc-093f8a1d9c08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941969 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.941990 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.942209 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home" pod "aa169038-e9bd-47c5-b6dc-093f8a1d9c08" (UID: "aa169038-e9bd-47c5-b6dc-093f8a1d9c08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.953290 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m" pod "aa169038-e9bd-47c5-b6dc-093f8a1d9c08" (UID: "aa169038-e9bd-47c5-b6dc-093f8a1d9c08"). InnerVolumeSpecName "kube-api-access-rrq2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.953919 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm" pod "aa169038-e9bd-47c5-b6dc-093f8a1d9c08" (UID: "aa169038-e9bd-47c5-b6dc-093f8a1d9c08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:14.954323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.953919 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs" pod "aa169038-e9bd-47c5-b6dc-093f8a1d9c08" (UID: "aa169038-e9bd-47c5-b6dc-093f8a1d9c08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:00:14.958509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.956978 4484 generic.go:356] "Generic (PLEG): container finished" podID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerID="443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c" exitCode=0 Nov 01 13:00:14.958509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.957035 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" Nov 01 13:00:14.958509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.957060 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" event={"ID":"aa169038-e9bd-47c5-b6dc-093f8a1d9c08","Type":"ContainerDied","Data":"443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c"} Nov 01 13:00:14.958509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.957106 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj" event={"ID":"aa169038-e9bd-47c5-b6dc-093f8a1d9c08","Type":"ContainerDied","Data":"2d673411835841fe001b6e18cc6cc2f64198daa15d0e804822a31e819c023d32"} Nov 01 13:00:14.958509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.957128 4484 scope.go:122] "RemoveContainer" containerID="443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c" Nov 01 13:00:14.966862 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.966835 4484 generic.go:356] "Generic (PLEG): container finished" podID="879f8d85-3a00-4665-8790-59fd23cffd53" containerID="29f700565167670445639e22911c9aa562ab7d658b418a21a5e70ab826b24030" exitCode=60 Nov 01 13:00:14.966994 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.966906 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerDied","Data":"29f700565167670445639e22911c9aa562ab7d658b418a21a5e70ab826b24030"} Nov 01 13:00:14.967483 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.967459 4484 scope.go:122] "RemoveContainer" containerID="29f700565167670445639e22911c9aa562ab7d658b418a21a5e70ab826b24030" Nov 01 13:00:14.967846 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:14.967823 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9_odh-ai-gateway-infra(879f8d85-3a00-4665-8790-59fd23cffd53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" Nov 01 13:00:14.983060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.983040 4484 scope.go:122] "RemoveContainer" containerID="443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c" Nov 01 13:00:14.983374 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:14.983338 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c\": container with ID starting with 443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c not found: ID does not exist" containerID="443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c" Nov 01 13:00:14.983495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.983380 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c"} err="failed to get container status \"443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c\": rpc error: code = NotFound desc = could not find container \"443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c\": container with ID starting with 443ae42c09c65828f577a4e855ec77528aa90a1056089f9fb95278c08f63950c not found: ID does not exist" Nov 01 13:00:14.983495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:14.983406 4484 scope.go:122] "RemoveContainer" containerID="d327d52ddfcc9383b5afcba2145b5898950bb55cc92910aefc7e2415c9425834" Nov 01 13:00:15.043576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.043535 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:15.043576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.043577 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrq2m\" (UniqueName: \"kubernetes.io/projected/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-kube-api-access-rrq2m\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:15.043847 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.043593 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:15.043847 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.043605 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa169038-e9bd-47c5-b6dc-093f8a1d9c08-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:15.057351 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.057314 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9"] Nov 01 13:00:15.070578 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.070551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:15.244486 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.244458 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj"] Nov 01 13:00:15.248509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.248483 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/simulator-w1-5b1bd4-kserve-5756b77b7b-4cmmj"] Nov 01 13:00:15.362593 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.362572 4484 scope.go:122] "RemoveContainer" containerID="181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129" Nov 01 13:00:15.362830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.362815 4484 scope.go:122] "RemoveContainer" containerID="8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5" Nov 01 13:00:15.882223 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.882143 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:00:15.975761 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.975727 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerStarted","Data":"84c81696c7f6345e3400ea9c8812ded2943e9af79920b3b2669f4f486e45c344"} Nov 01 13:00:15.978738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:15.978709 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerStarted","Data":"1c7116240d1ee8816ef190dd25736ca6e4df9a3ca0de488b885b29c40abfb890"} Nov 01 13:00:16.052212 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.051985 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerName="payload-processing" containerID="cri-o://950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283" gracePeriod=25 Nov 01 13:00:16.133895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.133835 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:16.158446 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.158416 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca\") pod \"879f8d85-3a00-4665-8790-59fd23cffd53\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " Nov 01 13:00:16.158634 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.158621 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl\" (UniqueName: \"kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl\") pod \"879f8d85-3a00-4665-8790-59fd23cffd53\" (UID: \"879f8d85-3a00-4665-8790-59fd23cffd53\") " Nov 01 13:00:16.158731 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.158709 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca" pod "879f8d85-3a00-4665-8790-59fd23cffd53" (UID: "879f8d85-3a00-4665-8790-59fd23cffd53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:16.158925 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.158911 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/879f8d85-3a00-4665-8790-59fd23cffd53-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:16.159987 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.159965 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerName="payload-pre-processing" containerID="cri-o://cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba" gracePeriod=25 Nov 01 13:00:16.160648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.160627 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl" pod "879f8d85-3a00-4665-8790-59fd23cffd53" (UID: "879f8d85-3a00-4665-8790-59fd23cffd53"). InnerVolumeSpecName "kube-api-access-m9fnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:16.260404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.260376 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fnl\" (UniqueName: \"kubernetes.io/projected/879f8d85-3a00-4665-8790-59fd23cffd53-kube-api-access-m9fnl\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:16.369291 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.369258 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" path="/var/lib/kubelet/pods/aa169038-e9bd-47c5-b6dc-093f8a1d9c08/volumes" Nov 01 13:00:16.480688 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.480657 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:00:16.889941 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.889856 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:00:16.939638 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.939610 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9"] Nov 01 13:00:16.940142 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940129 4484 cpu_manager.go:414] "containerMap: removing container" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940144 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940160 4484 cpu_manager.go:414] "containerMap: removing container" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940165 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940176 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerName="cleanup" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940181 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerName="cleanup" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940217 4484 cpu_manager.go:414] "containerMap: removing container" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:16.940220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940222 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:16.940465 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940322 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa169038-e9bd-47c5-b6dc-093f8a1d9c08" containerName="main" Nov 01 13:00:16.940465 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940333 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:16.940465 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940341 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerName="cleanup" Nov 01 13:00:16.940465 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940351 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerName="cleanup" Nov 01 13:00:16.940465 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.940358 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:16.944032 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.944013 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:16.968901 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.968878 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:16.969028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.969012 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvsqp\" (UniqueName: \"kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:16.969132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.969113 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:16.988828 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.988805 4484 generic.go:356] "Generic (PLEG): container finished" podID="a6493c38-f301-4c38-879d-6e7459179855" containerID="84c81696c7f6345e3400ea9c8812ded2943e9af79920b3b2669f4f486e45c344" exitCode=60 Nov 01 13:00:16.988932 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.988881 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerDied","Data":"84c81696c7f6345e3400ea9c8812ded2943e9af79920b3b2669f4f486e45c344"} Nov 01 13:00:16.988994 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.988940 4484 scope.go:122] "RemoveContainer" containerID="181e270626ffc5500068903b911c65dee58ad3c6af0a3232dd78b50ca68cb129" Nov 01 13:00:16.989199 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.989184 4484 scope.go:122] "RemoveContainer" containerID="84c81696c7f6345e3400ea9c8812ded2943e9af79920b3b2669f4f486e45c344" Nov 01 13:00:16.989793 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:16.989467 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md_odh-ai-gateway-infra(a6493c38-f301-4c38-879d-6e7459179855)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" podUID="a6493c38-f301-4c38-879d-6e7459179855" Nov 01 13:00:16.998484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.992073 4484 generic.go:356] "Generic (PLEG): container finished" podID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerID="1c7116240d1ee8816ef190dd25736ca6e4df9a3ca0de488b885b29c40abfb890" exitCode=60 Nov 01 13:00:16.998484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.992181 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerDied","Data":"1c7116240d1ee8816ef190dd25736ca6e4df9a3ca0de488b885b29c40abfb890"} Nov 01 13:00:16.998484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:16.992483 4484 scope.go:122] "RemoveContainer" containerID="1c7116240d1ee8816ef190dd25736ca6e4df9a3ca0de488b885b29c40abfb890" Nov 01 13:00:16.998484 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:16.992710 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn_odh-ai-gateway-infra(6c2fa7b7-8178-458c-b62b-e19675c4b2bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" Nov 01 13:00:17.016188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.016160 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:00:17.018697 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.018672 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" event={"ID":"879f8d85-3a00-4665-8790-59fd23cffd53","Type":"ContainerDied","Data":"43df97d0bdab03cb931296183b4539ecc96f49f51d3e6a3bdcce07d7ab46f576"} Nov 01 13:00:17.018920 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.018904 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9" Nov 01 13:00:17.019619 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.019603 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9"] Nov 01 13:00:17.030358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.030338 4484 scope.go:122] "RemoveContainer" containerID="8ba609b3c22ade8bdbecfdef64b5954c291b3eadeeb8ae19be6838ce8fe2e0b5" Nov 01 13:00:17.044456 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.044437 4484 scope.go:122] "RemoveContainer" containerID="29f700565167670445639e22911c9aa562ab7d658b418a21a5e70ab826b24030" Nov 01 13:00:17.070356 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.070325 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.071502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.070506 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.071502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.070667 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvsqp\" (UniqueName: \"kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.071502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.071122 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.081921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.081893 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.103424 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.103395 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md"] Nov 01 13:00:17.130335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.130292 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvsqp\" (UniqueName: \"kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp\") pod \"maas-api-revoke-sub-2635cf4f-x7xl9\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.256906 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.256873 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:17.315740 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.315715 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn"] Nov 01 13:00:17.362016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.361981 4484 scope.go:122] "RemoveContainer" containerID="6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15" Nov 01 13:00:17.439694 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.439661 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9"] Nov 01 13:00:17.442944 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.442919 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892300-g68m9"] Nov 01 13:00:17.631877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.631822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9"] Nov 01 13:00:17.937531 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:17.937498 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:18.031271 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.031223 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerStarted","Data":"c56bc02a1743dfb3d0fb813f27272d433cc5440f82882d2b7a277338aa14ec3f"} Nov 01 13:00:18.032626 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.032602 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" event={"ID":"7b03f64f-fe54-435f-a7f0-4e0533a4b776","Type":"ContainerStarted","Data":"cb1768accb31e4929196e1e50b55b74417d15d23d800b6ce963767f7f17546b5"} Nov 01 13:00:18.184903 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.184871 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:18.204816 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.204794 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:18.285971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.285946 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca\") pod \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " Nov 01 13:00:18.286132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.285996 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n\" (UniqueName: \"kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n\") pod \"a6493c38-f301-4c38-879d-6e7459179855\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " Nov 01 13:00:18.286132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.286087 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca\") pod \"a6493c38-f301-4c38-879d-6e7459179855\" (UID: \"a6493c38-f301-4c38-879d-6e7459179855\") " Nov 01 13:00:18.286255 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.286144 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j\" (UniqueName: \"kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j\") pod \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\" (UID: \"6c2fa7b7-8178-458c-b62b-e19675c4b2bf\") " Nov 01 13:00:18.286335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.286254 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca" pod "6c2fa7b7-8178-458c-b62b-e19675c4b2bf" (UID: "6c2fa7b7-8178-458c-b62b-e19675c4b2bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:18.286491 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.286471 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca" pod "a6493c38-f301-4c38-879d-6e7459179855" (UID: "a6493c38-f301-4c38-879d-6e7459179855"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:18.286625 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.286610 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:18.286669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.286627 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6493c38-f301-4c38-879d-6e7459179855-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:18.288066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.288045 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n" pod "a6493c38-f301-4c38-879d-6e7459179855" (UID: "a6493c38-f301-4c38-879d-6e7459179855"). InnerVolumeSpecName "kube-api-access-6kn6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:18.288141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.288121 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j" pod "6c2fa7b7-8178-458c-b62b-e19675c4b2bf" (UID: "6c2fa7b7-8178-458c-b62b-e19675c4b2bf"). InnerVolumeSpecName "kube-api-access-zjx4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:18.362773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.362746 4484 scope.go:122] "RemoveContainer" containerID="29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479" Nov 01 13:00:18.370106 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.370059 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" path="/var/lib/kubelet/pods/879f8d85-3a00-4665-8790-59fd23cffd53/volumes" Nov 01 13:00:18.388059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.388034 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjx4j\" (UniqueName: \"kubernetes.io/projected/6c2fa7b7-8178-458c-b62b-e19675c4b2bf-kube-api-access-zjx4j\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:18.388173 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:18.388154 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kn6n\" (UniqueName: \"kubernetes.io/projected/a6493c38-f301-4c38-879d-6e7459179855-kube-api-access-6kn6n\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:19.045074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.045039 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerStarted","Data":"ef2d8c6a3556ea7ec154f71e0600b96f24479f2729c33d291405c800829d9c81"} Nov 01 13:00:19.046538 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.046520 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" Nov 01 13:00:19.046625 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.046542 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md" event={"ID":"a6493c38-f301-4c38-879d-6e7459179855","Type":"ContainerDied","Data":"b69be058bce5e9f7a44d854012c916a321f7b31f8689926eab90902528bf09d0"} Nov 01 13:00:19.046625 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.046604 4484 scope.go:122] "RemoveContainer" containerID="84c81696c7f6345e3400ea9c8812ded2943e9af79920b3b2669f4f486e45c344" Nov 01 13:00:19.048753 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.048712 4484 generic.go:356] "Generic (PLEG): container finished" podID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerID="c56bc02a1743dfb3d0fb813f27272d433cc5440f82882d2b7a277338aa14ec3f" exitCode=60 Nov 01 13:00:19.048882 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.048814 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerDied","Data":"c56bc02a1743dfb3d0fb813f27272d433cc5440f82882d2b7a277338aa14ec3f"} Nov 01 13:00:19.049323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.049280 4484 scope.go:122] "RemoveContainer" containerID="c56bc02a1743dfb3d0fb813f27272d433cc5440f82882d2b7a277338aa14ec3f" Nov 01 13:00:19.049616 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:19.049586 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp_odh-ai-gateway-infra(bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" Nov 01 13:00:19.050806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.050784 4484 generic.go:356] "Generic (PLEG): container finished" podID="7b03f64f-fe54-435f-a7f0-4e0533a4b776" containerID="997d919a096d23008c6535535189a4a06e2c25bec2732b4109404bfc98510084" exitCode=0 Nov 01 13:00:19.050914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.050853 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" event={"ID":"7b03f64f-fe54-435f-a7f0-4e0533a4b776","Type":"ContainerDied","Data":"997d919a096d23008c6535535189a4a06e2c25bec2732b4109404bfc98510084"} Nov 01 13:00:19.052202 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.052180 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" event={"ID":"6c2fa7b7-8178-458c-b62b-e19675c4b2bf","Type":"ContainerDied","Data":"634c54d81e6b5c859c908cfeecfc5894480ac2593869e52d961a09108ac9b331"} Nov 01 13:00:19.052292 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.052206 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn" Nov 01 13:00:19.057554 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.057533 4484 scope.go:122] "RemoveContainer" containerID="6314dd9ceb212e97caff07f29e00e4660eaa8dd1c79581c5d1276bcbc77dfe15" Nov 01 13:00:19.072566 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.072548 4484 scope.go:122] "RemoveContainer" containerID="1c7116240d1ee8816ef190dd25736ca6e4df9a3ca0de488b885b29c40abfb890" Nov 01 13:00:19.361178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.361093 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp"] Nov 01 13:00:19.470570 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.470534 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md"] Nov 01 13:00:19.480501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.480476 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892300-cl6md"] Nov 01 13:00:19.581043 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.581015 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn"] Nov 01 13:00:19.584571 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:19.584544 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892300-ltwkn"] Nov 01 13:00:20.036540 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.036486 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 13:00:20.061016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.060992 4484 generic.go:356] "Generic (PLEG): container finished" podID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerID="ef2d8c6a3556ea7ec154f71e0600b96f24479f2729c33d291405c800829d9c81" exitCode=60 Nov 01 13:00:20.061398 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.061081 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerDied","Data":"ef2d8c6a3556ea7ec154f71e0600b96f24479f2729c33d291405c800829d9c81"} Nov 01 13:00:20.061398 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.061153 4484 scope.go:122] "RemoveContainer" containerID="29fa36aa32a10cca74d88c749111d7a5a95f3ab5a84338571b1394d813210479" Nov 01 13:00:20.061545 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.061530 4484 scope.go:122] "RemoveContainer" containerID="ef2d8c6a3556ea7ec154f71e0600b96f24479f2729c33d291405c800829d9c81" Nov 01 13:00:20.061854 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:20.061826 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf_odh-ai-gateway-infra(b1a4ab08-5544-4d77-bcb6-efe88470f9ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" Nov 01 13:00:20.063108 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.063090 4484 generic.go:356] "Generic (PLEG): container finished" podID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerID="7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813" exitCode=0 Nov 01 13:00:20.063183 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.063127 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" Nov 01 13:00:20.063229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.063174 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" event={"ID":"10a57d05-e15f-4aed-b79f-3245af6376a9","Type":"ContainerDied","Data":"7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813"} Nov 01 13:00:20.063282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.063227 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc" event={"ID":"10a57d05-e15f-4aed-b79f-3245af6376a9","Type":"ContainerDied","Data":"9ba73cf31b4237d1eac460183915f8aad9fc8c3c818616a87310bed36830fb00"} Nov 01 13:00:20.081612 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.080514 4484 scope.go:122] "RemoveContainer" containerID="7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813" Nov 01 13:00:20.092314 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.092279 4484 scope.go:122] "RemoveContainer" containerID="662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1" Nov 01 13:00:20.105895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.105872 4484 scope.go:122] "RemoveContainer" containerID="7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813" Nov 01 13:00:20.106226 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:20.106198 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813\": container with ID starting with 7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813 not found: ID does not exist" containerID="7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813" Nov 01 13:00:20.106326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.106234 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813"} err="failed to get container status \"7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813\": rpc error: code = NotFound desc = could not find container \"7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813\": container with ID starting with 7bba05c01295ee314788bfcbad166f0d58819eee37ffa6c5598b5f86af095813 not found: ID does not exist" Nov 01 13:00:20.106326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.106253 4484 scope.go:122] "RemoveContainer" containerID="662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1" Nov 01 13:00:20.106567 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:20.106542 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1\": container with ID starting with 662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1 not found: ID does not exist" containerID="662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1" Nov 01 13:00:20.106615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.106577 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1"} err="failed to get container status \"662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1\": rpc error: code = NotFound desc = could not find container \"662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1\": container with ID starting with 662f48c1c7882d63359acf355dfe59999ea2b4c8540fa20b7f6f96c34ffca2d1 not found: ID does not exist" Nov 01 13:00:20.108648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108616 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108718 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108683 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108768 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108725 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108768 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108762 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108835 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108781 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108835 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108805 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108836 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108889 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.108965 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.108912 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket\") pod \"10a57d05-e15f-4aed-b79f-3245af6376a9\" (UID: \"10a57d05-e15f-4aed-b79f-3245af6376a9\") " Nov 01 13:00:20.109597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.109477 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:20.110121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.109857 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:20.110121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.109892 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:20.110121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.109991 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:20.110121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.110083 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:20.112620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.112483 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:00:20.112620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.112575 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:20.112620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.112594 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "kube-api-access-vswkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:20.112620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.112601 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy" pod "10a57d05-e15f-4aed-b79f-3245af6376a9" (UID: "10a57d05-e15f-4aed-b79f-3245af6376a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:00:20.165916 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.165885 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf"] Nov 01 13:00:20.210273 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210241 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vswkh\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-kube-api-access-vswkh\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210278 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210296 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210327 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210341 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a57d05-e15f-4aed-b79f-3245af6376a9-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210354 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210369 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210384 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.210441 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.210396 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a57d05-e15f-4aed-b79f-3245af6376a9-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.214150 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.214129 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:20.257000 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.256977 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:20.310864 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.310797 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca\") pod \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " Nov 01 13:00:20.310971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.310924 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp\" (UniqueName: \"kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp\") pod \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " Nov 01 13:00:20.311038 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.310992 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca\") pod \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " Nov 01 13:00:20.311076 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.311040 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx\" (UniqueName: \"kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx\") pod \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\" (UID: \"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a\") " Nov 01 13:00:20.311120 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.311101 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca\") pod \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\" (UID: \"7b03f64f-fe54-435f-a7f0-4e0533a4b776\") " Nov 01 13:00:20.311367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.311332 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca" pod "bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" (UID: "bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:20.311587 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.311564 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.311845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.311823 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca" pod "7b03f64f-fe54-435f-a7f0-4e0533a4b776" (UID: "7b03f64f-fe54-435f-a7f0-4e0533a4b776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:20.312957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.312935 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca" pod "7b03f64f-fe54-435f-a7f0-4e0533a4b776" (UID: "7b03f64f-fe54-435f-a7f0-4e0533a4b776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:00:20.313152 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.313132 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx" pod "bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" (UID: "bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a"). InnerVolumeSpecName "kube-api-access-zsgxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:20.313213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.313163 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp" pod "7b03f64f-fe54-435f-a7f0-4e0533a4b776" (UID: "7b03f64f-fe54-435f-a7f0-4e0533a4b776"). InnerVolumeSpecName "kube-api-access-dvsqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:20.378843 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.378816 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" path="/var/lib/kubelet/pods/6c2fa7b7-8178-458c-b62b-e19675c4b2bf/volumes" Nov 01 13:00:20.379238 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.379226 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6493c38-f301-4c38-879d-6e7459179855" path="/var/lib/kubelet/pods/a6493c38-f301-4c38-879d-6e7459179855/volumes" Nov 01 13:00:20.412591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.412562 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.412591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.412587 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvsqp\" (UniqueName: \"kubernetes.io/projected/7b03f64f-fe54-435f-a7f0-4e0533a4b776-kube-api-access-dvsqp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.412740 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.412604 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgxx\" (UniqueName: \"kubernetes.io/projected/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a-kube-api-access-zsgxx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.412740 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.412617 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b03f64f-fe54-435f-a7f0-4e0533a4b776-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:20.721966 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.721884 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc"] Nov 01 13:00:20.725810 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:20.725781 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-94c13a-openshift-default-6dfc756bbc-vcxbc"] Nov 01 13:00:21.073921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.073893 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" Nov 01 13:00:21.074397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.073893 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9" event={"ID":"7b03f64f-fe54-435f-a7f0-4e0533a4b776","Type":"ContainerDied","Data":"cb1768accb31e4929196e1e50b55b74417d15d23d800b6ce963767f7f17546b5"} Nov 01 13:00:21.074397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.074022 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1768accb31e4929196e1e50b55b74417d15d23d800b6ce963767f7f17546b5" Nov 01 13:00:21.079655 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.079630 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" event={"ID":"bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a","Type":"ContainerDied","Data":"820fde903e53d3028727e8d6e0e9d53c15782773ac405519de00a7beedbdfab6"} Nov 01 13:00:21.079784 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.079678 4484 scope.go:122] "RemoveContainer" containerID="c56bc02a1743dfb3d0fb813f27272d433cc5440f82882d2b7a277338aa14ec3f" Nov 01 13:00:21.079784 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.079638 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp" Nov 01 13:00:21.225993 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.225963 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:21.284900 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.284868 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp"] Nov 01 13:00:21.288381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.288360 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-187d06ed-29892300-rnxhp"] Nov 01 13:00:21.322176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.322148 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca\") pod \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " Nov 01 13:00:21.322320 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.322219 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp\" (UniqueName: \"kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp\") pod \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\" (UID: \"b1a4ab08-5544-4d77-bcb6-efe88470f9ec\") " Nov 01 13:00:21.322538 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.322518 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca" pod "b1a4ab08-5544-4d77-bcb6-efe88470f9ec" (UID: "b1a4ab08-5544-4d77-bcb6-efe88470f9ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:21.324282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.324234 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp" pod "b1a4ab08-5544-4d77-bcb6-efe88470f9ec" (UID: "b1a4ab08-5544-4d77-bcb6-efe88470f9ec"). InnerVolumeSpecName "kube-api-access-whpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:21.424130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.424100 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:21.424130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.424124 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpmp\" (UniqueName: \"kubernetes.io/projected/b1a4ab08-5544-4d77-bcb6-efe88470f9ec-kube-api-access-whpmp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:21.982868 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.982834 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt"] Nov 01 13:00:21.983347 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983334 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:21.983397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983348 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:21.983397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983359 4484 cpu_manager.go:414] "containerMap: removing container" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" Nov 01 13:00:21.983397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983364 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" Nov 01 13:00:21.983397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983388 4484 cpu_manager.go:414] "containerMap: removing container" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:21.983397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983393 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983400 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983406 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983413 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983417 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983425 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7b03f64f-fe54-435f-a7f0-4e0533a4b776" containerName="revoke-keys" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983430 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b03f64f-fe54-435f-a7f0-4e0533a4b776" containerName="revoke-keys" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983439 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983444 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983449 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983453 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983459 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983464 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983472 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983476 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983483 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983488 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983496 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983501 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983508 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983512 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983521 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerName="cleanup" Nov 01 13:00:21.983547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983525 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabbbce6-c46c-459f-99ef-6fce9ca2f4b0" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983603 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b03f64f-fe54-435f-a7f0-4e0533a4b776" containerName="revoke-keys" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983613 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983619 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983624 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983630 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983635 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983640 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983645 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="879f8d85-3a00-4665-8790-59fd23cffd53" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983650 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983654 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983661 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983667 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:21.984170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.983674 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:21.987110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:21.987089 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.029671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.029644 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.029839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.029701 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.029894 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.029871 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqd9c\" (UniqueName: \"kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.036421 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.036397 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:00:22.040242 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.040221 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt"] Nov 01 13:00:22.087319 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.087278 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" Nov 01 13:00:22.087711 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.087321 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf" event={"ID":"b1a4ab08-5544-4d77-bcb6-efe88470f9ec","Type":"ContainerDied","Data":"c47f4eeb17fe48e546115c0fe5cc07dbffb395bab02fff68c93a540938e148f9"} Nov 01 13:00:22.087711 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.087382 4484 scope.go:122] "RemoveContainer" containerID="ef2d8c6a3556ea7ec154f71e0600b96f24479f2729c33d291405c800829d9c81" Nov 01 13:00:22.131425 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.131400 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqd9c\" (UniqueName: \"kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.131536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.131509 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.131588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.131556 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.132126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.132101 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.133740 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.133720 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.184000 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.183973 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqd9c\" (UniqueName: \"kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c\") pod \"maas-api-revoke-sub-a60878d2-rtvxt\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.243443 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.243391 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf"] Nov 01 13:00:22.247680 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.247655 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ecd640-29892300-mv4tf"] Nov 01 13:00:22.300804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.300783 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:22.378086 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.378052 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" path="/var/lib/kubelet/pods/10a57d05-e15f-4aed-b79f-3245af6376a9/volumes" Nov 01 13:00:22.378713 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.378691 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a4ab08-5544-4d77-bcb6-efe88470f9ec" path="/var/lib/kubelet/pods/b1a4ab08-5544-4d77-bcb6-efe88470f9ec/volumes" Nov 01 13:00:22.379215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.379195 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" path="/var/lib/kubelet/pods/bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a/volumes" Nov 01 13:00:22.665924 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:22.665889 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt"] Nov 01 13:00:22.667731 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:22.667705 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad86fe5_f9d4_4452_ad36_cc00679df786.slice/crio-221c2b49b7adc705128560dc01178dac6d6012d8b235aca362a2c564d1dddb36 WatchSource:0}: Error finding container 221c2b49b7adc705128560dc01178dac6d6012d8b235aca362a2c564d1dddb36: Status 404 returned error can't find the container with id 221c2b49b7adc705128560dc01178dac6d6012d8b235aca362a2c564d1dddb36 Nov 01 13:00:23.096872 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:23.096830 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" event={"ID":"2ad86fe5-f9d4-4452-ad36-cc00679df786","Type":"ContainerStarted","Data":"022b20c0b7feb58c30e95e5a7b024d606c9d7ab53b97a1e0d36a746df2e625c5"} Nov 01 13:00:23.097283 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:23.096920 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" event={"ID":"2ad86fe5-f9d4-4452-ad36-cc00679df786","Type":"ContainerStarted","Data":"221c2b49b7adc705128560dc01178dac6d6012d8b235aca362a2c564d1dddb36"} Nov 01 13:00:23.202088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:23.201994 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" podStartSLOduration=2.201981964 podStartE2EDuration="2.201981964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:23.201585265 +0000 UTC m=+2292.951528255" watchObservedRunningTime="2026-11-01 13:00:23.201981964 +0000 UTC m=+2292.951924941" Nov 01 13:00:23.792533 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:23.792501 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:24.106724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:24.106634 4484 generic.go:356] "Generic (PLEG): container finished" podID="2ad86fe5-f9d4-4452-ad36-cc00679df786" containerID="022b20c0b7feb58c30e95e5a7b024d606c9d7ab53b97a1e0d36a746df2e625c5" exitCode=0 Nov 01 13:00:24.106724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:24.106711 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" event={"ID":"2ad86fe5-f9d4-4452-ad36-cc00679df786","Type":"ContainerDied","Data":"022b20c0b7feb58c30e95e5a7b024d606c9d7ab53b97a1e0d36a746df2e625c5"} Nov 01 13:00:25.248180 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.248158 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:25.369493 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.369407 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca\") pod \"2ad86fe5-f9d4-4452-ad36-cc00679df786\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " Nov 01 13:00:25.369650 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.369540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca\") pod \"2ad86fe5-f9d4-4452-ad36-cc00679df786\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " Nov 01 13:00:25.369732 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.369710 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c\" (UniqueName: \"kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c\") pod \"2ad86fe5-f9d4-4452-ad36-cc00679df786\" (UID: \"2ad86fe5-f9d4-4452-ad36-cc00679df786\") " Nov 01 13:00:25.369874 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.369847 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca" pod "2ad86fe5-f9d4-4452-ad36-cc00679df786" (UID: "2ad86fe5-f9d4-4452-ad36-cc00679df786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:25.370099 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.370076 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:25.371591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.371563 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca" pod "2ad86fe5-f9d4-4452-ad36-cc00679df786" (UID: "2ad86fe5-f9d4-4452-ad36-cc00679df786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:00:25.371725 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.371701 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c" pod "2ad86fe5-f9d4-4452-ad36-cc00679df786" (UID: "2ad86fe5-f9d4-4452-ad36-cc00679df786"). InnerVolumeSpecName "kube-api-access-jqd9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:25.471139 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.471113 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqd9c\" (UniqueName: \"kubernetes.io/projected/2ad86fe5-f9d4-4452-ad36-cc00679df786-kube-api-access-jqd9c\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:25.471275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.471144 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad86fe5-f9d4-4452-ad36-cc00679df786-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:25.475664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:25.475636 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:00:26.121096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:26.121058 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" Nov 01 13:00:26.121268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:26.121103 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt" event={"ID":"2ad86fe5-f9d4-4452-ad36-cc00679df786","Type":"ContainerDied","Data":"221c2b49b7adc705128560dc01178dac6d6012d8b235aca362a2c564d1dddb36"} Nov 01 13:00:26.121268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:26.121158 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221c2b49b7adc705128560dc01178dac6d6012d8b235aca362a2c564d1dddb36" Nov 01 13:00:26.480643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:26.480603 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:00:27.199707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:27.199669 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv"] Nov 01 13:00:27.301823 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:27.301790 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x"] Nov 01 13:00:27.544105 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:27.544073 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:27.659912 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:27.659873 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:28.328976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.328942 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:28.465252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.465223 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:28.560026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.559992 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76"] Nov 01 13:00:28.560576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560560 4484 cpu_manager.go:414] "containerMap: removing container" podUID="2ad86fe5-f9d4-4452-ad36-cc00679df786" containerName="revoke-keys" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560578 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad86fe5-f9d4-4452-ad36-cc00679df786" containerName="revoke-keys" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560596 4484 cpu_manager.go:414] "containerMap: removing container" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560601 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a57d05-e15f-4aed-b79f-3245af6376a9" containerName="istio-proxy" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560632 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560638 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560647 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:28.560649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560651 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:28.560860 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560733 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5cad83-6f70-40bb-ba6b-e7da46fe3a3a" containerName="cleanup" Nov 01 13:00:28.560860 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560743 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2fa7b7-8178-458c-b62b-e19675c4b2bf" containerName="cleanup" Nov 01 13:00:28.560860 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560752 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6493c38-f301-4c38-879d-6e7459179855" containerName="cleanup" Nov 01 13:00:28.560860 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.560758 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad86fe5-f9d4-4452-ad36-cc00679df786" containerName="revoke-keys" Nov 01 13:00:28.564293 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.564272 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.606115 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.606052 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh"] Nov 01 13:00:28.609938 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.609922 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.612604 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.612581 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-187d06ed-openshift-default-dockercfg-h6ggd\"" Nov 01 13:00:28.729080 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.729035 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.732648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.732616 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.732978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.732883 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.732978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.732933 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.755286 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755250 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.755515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755331 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.755515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755362 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.755515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755387 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.755515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755428 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.755515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755467 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.755806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755575 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.755806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755774 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.755911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755820 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.755911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755847 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.755911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755879 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.756066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.755937 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbld\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.756066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.756009 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpllx\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.756066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.756038 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.758935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.758910 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh"] Nov 01 13:00:28.762962 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.762934 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76"] Nov 01 13:00:28.857509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.857349 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpllx\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.857776 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.857748 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.857938 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.857923 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.858058 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858042 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.858155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858142 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.858259 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858246 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.858391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858370 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.858526 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858510 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.858655 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858639 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.858761 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858745 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.858887 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.858870 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.859028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.859007 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.861129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861096 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.861246 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.860409 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.861246 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.860682 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.861451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861426 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.861532 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.859529 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.861674 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861547 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.861674 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861608 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.861674 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861657 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.861826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861686 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.861826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861724 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.861826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.861795 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbld\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.862423 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.862401 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.863150 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.863127 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.864026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.863946 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.871617 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.871561 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.872030 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.872008 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.898897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.898864 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.899163 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.899138 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.899530 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.899498 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.903368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.903347 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.912670 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.912644 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpllx\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.913081 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.913061 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:28.913437 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.913415 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.914242 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.914217 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbld\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:28.925767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:28.925743 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:29.176926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.176835 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:29.205350 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.205315 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:29.392609 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.392586 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh"] Nov 01 13:00:29.394500 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:29.394461 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0cd339_c08e_40c9_a5a1_7ba0e791b0c5.slice/crio-899046614c14be313e614650116cf27045520ed89d81cacd429fdf7b3bb1e974 WatchSource:0}: Error finding container 899046614c14be313e614650116cf27045520ed89d81cacd429fdf7b3bb1e974: Status 404 returned error can't find the container with id 899046614c14be313e614650116cf27045520ed89d81cacd429fdf7b3bb1e974 Nov 01 13:00:29.396986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.396955 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:29.397059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.397021 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:29.397059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.397048 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:29.547660 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.547631 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76"] Nov 01 13:00:29.549272 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:29.549247 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387f169d_f247_488d_851d_509b9a889943.slice/crio-b2c2ceca404779736c5545a64a182cd9fc6121ca800d96fd9ce5307d55af904d WatchSource:0}: Error finding container b2c2ceca404779736c5545a64a182cd9fc6121ca800d96fd9ce5307d55af904d: Status 404 returned error can't find the container with id b2c2ceca404779736c5545a64a182cd9fc6121ca800d96fd9ce5307d55af904d Nov 01 13:00:29.551996 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.551959 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:29.552096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.552032 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:29.552096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:29.552058 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:30.200903 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.200856 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" event={"ID":"387f169d-f247-488d-851d-509b9a889943","Type":"ContainerStarted","Data":"e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa"} Nov 01 13:00:30.201777 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.200964 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" event={"ID":"387f169d-f247-488d-851d-509b9a889943","Type":"ContainerStarted","Data":"b2c2ceca404779736c5545a64a182cd9fc6121ca800d96fd9ce5307d55af904d"} Nov 01 13:00:30.203534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.203502 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" event={"ID":"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5","Type":"ContainerStarted","Data":"15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b"} Nov 01 13:00:30.203682 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.203567 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" event={"ID":"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5","Type":"ContainerStarted","Data":"899046614c14be313e614650116cf27045520ed89d81cacd429fdf7b3bb1e974"} Nov 01 13:00:30.300418 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:30.299438 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:00:30.320547 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.320502 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podStartSLOduration=2.320488473 podStartE2EDuration="2.320488473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:30.319709209 +0000 UTC m=+2300.069652187" watchObservedRunningTime="2026-11-01 13:00:30.320488473 +0000 UTC m=+2300.070431445" Nov 01 13:00:30.465381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.464426 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podStartSLOduration=2.464407286 podStartE2EDuration="2.464407286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:30.460857909 +0000 UTC m=+2300.210800926" watchObservedRunningTime="2026-11-01 13:00:30.464407286 +0000 UTC m=+2300.214350265" Nov 01 13:00:30.926227 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.926190 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:30.927827 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.927806 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 01 13:00:30.927880 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:30.927851 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 01 13:00:31.181384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:31.181137 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:31.181384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:31.181266 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 13:00:31.181384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:31.181354 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 13:00:31.926707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:31.926675 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 01 13:00:31.927108 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:31.926730 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 01 13:00:32.178821 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.178726 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 13:00:32.179139 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.179096 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 13:00:32.200850 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.200800 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" podUID="cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" containerName="payload-processing" containerID="cri-o://4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20" gracePeriod=25 Nov 01 13:00:32.302074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.302029 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" podUID="6b169ce6-fd56-4dd5-ac66-747913843465" containerName="payload-pre-processing" containerID="cri-o://8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320" gracePeriod=25 Nov 01 13:00:32.606477 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.606136 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 13:00:32.626715 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.626676 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw\" (UniqueName: \"kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw\") pod \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " Nov 01 13:00:32.626861 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.626749 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume\") pod \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\" (UID: \"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac\") " Nov 01 13:00:32.627748 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.627715 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume" pod "cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" (UID: "cdcd8b6e-1628-4d90-aba1-fb9adc04bcac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:32.631671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.631645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw" pod "cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" (UID: "cdcd8b6e-1628-4d90-aba1-fb9adc04bcac"). InnerVolumeSpecName "kube-api-access-rrfrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:32.709558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.709532 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 13:00:32.724350 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.724319 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:32.727752 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.727732 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume\") pod \"6b169ce6-fd56-4dd5-ac66-747913843465\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " Nov 01 13:00:32.727817 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.727798 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss\" (UniqueName: \"kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss\") pod \"6b169ce6-fd56-4dd5-ac66-747913843465\" (UID: \"6b169ce6-fd56-4dd5-ac66-747913843465\") " Nov 01 13:00:32.728062 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.728044 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrfrw\" (UniqueName: \"kubernetes.io/projected/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-kube-api-access-rrfrw\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:32.728062 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.728058 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:32.728417 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.728392 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume" pod "6b169ce6-fd56-4dd5-ac66-747913843465" (UID: "6b169ce6-fd56-4dd5-ac66-747913843465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:32.729902 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.729882 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss" pod "6b169ce6-fd56-4dd5-ac66-747913843465" (UID: "6b169ce6-fd56-4dd5-ac66-747913843465"). InnerVolumeSpecName "kube-api-access-npkss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:32.833230 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.833141 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b169ce6-fd56-4dd5-ac66-747913843465-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:32.833230 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.833170 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npkss\" (UniqueName: \"kubernetes.io/projected/6b169ce6-fd56-4dd5-ac66-747913843465-kube-api-access-npkss\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:32.926906 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.926874 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 01 13:00:32.927281 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:32.926921 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 01 13:00:33.181801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.181704 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 13:00:33.182102 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.182063 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 13:00:33.223879 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.223852 4484 generic.go:356] "Generic (PLEG): container finished" podID="cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" containerID="4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20" exitCode=0 Nov 01 13:00:33.224064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.224047 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" Nov 01 13:00:33.224564 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.223982 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" event={"ID":"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac","Type":"ContainerDied","Data":"4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20"} Nov 01 13:00:33.224750 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.224728 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv" event={"ID":"cdcd8b6e-1628-4d90-aba1-fb9adc04bcac","Type":"ContainerDied","Data":"f62a475dd383fef394b9068e7033d35000f5381ec71e9abaf4889de48880e45f"} Nov 01 13:00:33.224877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.224863 4484 scope.go:122] "RemoveContainer" containerID="4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20" Nov 01 13:00:33.227667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.226864 4484 generic.go:356] "Generic (PLEG): container finished" podID="6b169ce6-fd56-4dd5-ac66-747913843465" containerID="8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320" exitCode=0 Nov 01 13:00:33.227667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.227002 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" event={"ID":"6b169ce6-fd56-4dd5-ac66-747913843465","Type":"ContainerDied","Data":"8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320"} Nov 01 13:00:33.227667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.227040 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" event={"ID":"6b169ce6-fd56-4dd5-ac66-747913843465","Type":"ContainerDied","Data":"c96742a2852fb53ae83d8966efe6bf6b70d785b8ce0eb0fe0239b6802b098ba2"} Nov 01 13:00:33.227667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.227112 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x" Nov 01 13:00:33.236526 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.236509 4484 scope.go:122] "RemoveContainer" containerID="4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20" Nov 01 13:00:33.236806 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:33.236773 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20\": container with ID starting with 4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20 not found: ID does not exist" containerID="4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20" Nov 01 13:00:33.236848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.236814 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20"} err="failed to get container status \"4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20\": rpc error: code = NotFound desc = could not find container \"4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20\": container with ID starting with 4d95a8ea222f27951758698d4baedf66271ddd07b1556c36491a41f351b1dc20 not found: ID does not exist" Nov 01 13:00:33.236848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.236830 4484 scope.go:122] "RemoveContainer" containerID="8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320" Nov 01 13:00:33.302461 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.301577 4484 scope.go:122] "RemoveContainer" containerID="8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320" Nov 01 13:00:33.302461 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:33.302015 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320\": container with ID starting with 8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320 not found: ID does not exist" containerID="8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320" Nov 01 13:00:33.302461 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.302058 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320"} err="failed to get container status \"8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320\": rpc error: code = NotFound desc = could not find container \"8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320\": container with ID starting with 8716dcf32203dd1b9174724aaeedc3f0ba8441bf12609ee21ff5637cfab42320 not found: ID does not exist" Nov 01 13:00:33.392446 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.392385 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x"] Nov 01 13:00:33.395067 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.395040 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2810c2-64f5c5dcf9-cwq4x"] Nov 01 13:00:33.497890 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.497855 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv"] Nov 01 13:00:33.503580 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.502264 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2810c2-6bd4d76cd5-mgsdv"] Nov 01 13:00:33.840425 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.838732 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:33.926229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.926194 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 01 13:00:33.926436 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:33.926246 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 01 13:00:34.179794 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:34.179709 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 13:00:34.180374 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:34.180340 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 13:00:34.421402 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:34.421184 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b169ce6-fd56-4dd5-ac66-747913843465" path="/var/lib/kubelet/pods/6b169ce6-fd56-4dd5-ac66-747913843465/volumes" Nov 01 13:00:34.421661 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:34.421644 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" path="/var/lib/kubelet/pods/cdcd8b6e-1628-4d90-aba1-fb9adc04bcac/volumes" Nov 01 13:00:34.926612 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:34.926575 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 01 13:00:34.926767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:34.926627 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 01 13:00:35.177739 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.177647 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 13:00:35.178141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.178110 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 13:00:35.475265 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.475175 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:00:35.538743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.538712 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:35.807327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.807268 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:35.933827 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.933795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:35.983055 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.983006 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:35.983591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.983544 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:35.984714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:35.984694 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:00:36.116399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.116367 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c"] Nov 01 13:00:36.118986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.118959 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-94c94b-q567c"] Nov 01 13:00:36.207790 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.207636 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:36.210327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.208466 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:36.210327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.210014 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:00:36.372775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.372751 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 13:00:36.373669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.373641 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce6085e-963c-453f-8acc-5d1f6bafc523" path="/var/lib/kubelet/pods/7ce6085e-963c-453f-8acc-5d1f6bafc523/volumes" Nov 01 13:00:36.423248 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.423227 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 13:00:36.427522 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.427491 4484 generic.go:356] "Generic (PLEG): container finished" podID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerID="950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283" exitCode=0 Nov 01 13:00:36.427671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.427567 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" Nov 01 13:00:36.427671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.427578 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" event={"ID":"24d7c973-1fdd-44ed-92a1-24651716dfef","Type":"ContainerDied","Data":"950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283"} Nov 01 13:00:36.427671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.427641 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz" event={"ID":"24d7c973-1fdd-44ed-92a1-24651716dfef","Type":"ContainerDied","Data":"18d5f072f0626548e80a72e68edc08d63c94babbf6220f2d141375c76d338864"} Nov 01 13:00:36.427830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.427674 4484 scope.go:122] "RemoveContainer" containerID="950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283" Nov 01 13:00:36.429066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.429045 4484 generic.go:356] "Generic (PLEG): container finished" podID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerID="cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba" exitCode=0 Nov 01 13:00:36.429188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.429096 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" Nov 01 13:00:36.429188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.429132 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" event={"ID":"80f666c7-0692-4a6b-bb77-7d8bdda8e342","Type":"ContainerDied","Data":"cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba"} Nov 01 13:00:36.429188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.429177 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj" event={"ID":"80f666c7-0692-4a6b-bb77-7d8bdda8e342","Type":"ContainerDied","Data":"e241db8c5a59a5d2d9dc0f3e2d68c1a8fd41164e5879d584d4a630f3df7d5241"} Nov 01 13:00:36.439238 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.439216 4484 scope.go:122] "RemoveContainer" containerID="950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283" Nov 01 13:00:36.439569 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:36.439530 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283\": container with ID starting with 950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283 not found: ID does not exist" containerID="950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283" Nov 01 13:00:36.439642 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.439572 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283"} err="failed to get container status \"950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283\": rpc error: code = NotFound desc = could not find container \"950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283\": container with ID starting with 950c633cf40c101f85f71aa3de7d9bc3e0ffdb31262c7c23c62637b3de65e283 not found: ID does not exist" Nov 01 13:00:36.439642 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.439595 4484 scope.go:122] "RemoveContainer" containerID="cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba" Nov 01 13:00:36.450658 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.450634 4484 scope.go:122] "RemoveContainer" containerID="cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba" Nov 01 13:00:36.450935 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:00:36.450914 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba\": container with ID starting with cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba not found: ID does not exist" containerID="cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba" Nov 01 13:00:36.451033 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.450941 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba"} err="failed to get container status \"cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba\": rpc error: code = NotFound desc = could not find container \"cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba\": container with ID starting with cbecf9ab73af95aeafa7cc453ebe97745c39eb9a56106a4165439be413c7e3ba not found: ID does not exist" Nov 01 13:00:36.455395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.455368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf\" (UniqueName: \"kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf\") pod \"24d7c973-1fdd-44ed-92a1-24651716dfef\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " Nov 01 13:00:36.455525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.455508 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume\") pod \"24d7c973-1fdd-44ed-92a1-24651716dfef\" (UID: \"24d7c973-1fdd-44ed-92a1-24651716dfef\") " Nov 01 13:00:36.455928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.455903 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume" pod "24d7c973-1fdd-44ed-92a1-24651716dfef" (UID: "24d7c973-1fdd-44ed-92a1-24651716dfef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:36.456094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.456072 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d7c973-1fdd-44ed-92a1-24651716dfef-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:36.457671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.457647 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf" pod "24d7c973-1fdd-44ed-92a1-24651716dfef" (UID: "24d7c973-1fdd-44ed-92a1-24651716dfef"). InnerVolumeSpecName "kube-api-access-69sqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:36.469051 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.469021 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:36.556571 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.556543 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume\") pod \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " Nov 01 13:00:36.557053 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.556666 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6\" (UniqueName: \"kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6\") pod \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\" (UID: \"80f666c7-0692-4a6b-bb77-7d8bdda8e342\") " Nov 01 13:00:36.557053 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.556954 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume" pod "80f666c7-0692-4a6b-bb77-7d8bdda8e342" (UID: "80f666c7-0692-4a6b-bb77-7d8bdda8e342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:00:36.557446 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.557422 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69sqf\" (UniqueName: \"kubernetes.io/projected/24d7c973-1fdd-44ed-92a1-24651716dfef-kube-api-access-69sqf\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:36.557558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.557450 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f666c7-0692-4a6b-bb77-7d8bdda8e342-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:36.559080 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.559050 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6" pod "80f666c7-0692-4a6b-bb77-7d8bdda8e342" (UID: "80f666c7-0692-4a6b-bb77-7d8bdda8e342"). InnerVolumeSpecName "kube-api-access-q7zx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:00:36.659155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.659079 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7zx6\" (UniqueName: \"kubernetes.io/projected/80f666c7-0692-4a6b-bb77-7d8bdda8e342-kube-api-access-q7zx6\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:00:36.844983 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.844950 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz"] Nov 01 13:00:36.850247 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.850220 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b1bd4-67bbcb6b8f-w2lqz"] Nov 01 13:00:36.957764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.957675 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj"] Nov 01 13:00:36.973020 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:36.972993 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b1bd4-fcb6f5f59-9qxrj"] Nov 01 13:00:37.039955 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:37.039925 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:37.276451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:37.276424 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:37.690155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:37.690019 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:38.369284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:38.369249 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" path="/var/lib/kubelet/pods/24d7c973-1fdd-44ed-92a1-24651716dfef/volumes" Nov 01 13:00:38.369717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:38.369702 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" path="/var/lib/kubelet/pods/80f666c7-0692-4a6b-bb77-7d8bdda8e342/volumes" Nov 01 13:00:38.978047 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:38.978012 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:41.593765 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:41.593730 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:42.067077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:42.067041 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:42.186651 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:42.186605 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:42.367886 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:42.367807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:43.349459 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.349424 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:43.664515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664437 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw"] Nov 01 13:00:43.664926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664914 4484 cpu_manager.go:414] "containerMap: removing container" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerName="payload-processing" Nov 01 13:00:43.664974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664927 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerName="payload-processing" Nov 01 13:00:43.664974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664938 4484 cpu_manager.go:414] "containerMap: removing container" podUID="cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" containerName="payload-processing" Nov 01 13:00:43.664974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664943 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" containerName="payload-processing" Nov 01 13:00:43.664974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664952 4484 cpu_manager.go:414] "containerMap: removing container" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerName="payload-pre-processing" Nov 01 13:00:43.664974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664957 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerName="payload-pre-processing" Nov 01 13:00:43.665129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664977 4484 cpu_manager.go:414] "containerMap: removing container" podUID="6b169ce6-fd56-4dd5-ac66-747913843465" containerName="payload-pre-processing" Nov 01 13:00:43.665129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.664982 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b169ce6-fd56-4dd5-ac66-747913843465" containerName="payload-pre-processing" Nov 01 13:00:43.665129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.665080 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d7c973-1fdd-44ed-92a1-24651716dfef" containerName="payload-processing" Nov 01 13:00:43.665129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.665087 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcd8b6e-1628-4d90-aba1-fb9adc04bcac" containerName="payload-processing" Nov 01 13:00:43.665129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.665094 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f666c7-0692-4a6b-bb77-7d8bdda8e342" containerName="payload-pre-processing" Nov 01 13:00:43.665129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.665104 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b169ce6-fd56-4dd5-ac66-747913843465" containerName="payload-pre-processing" Nov 01 13:00:43.668429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.668413 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695646 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695618 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695790 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695662 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2k7w\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695790 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695682 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695844 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695902 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695999 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695945 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.695999 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.695963 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.696086 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.696070 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.696136 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.696107 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797278 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797242 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797278 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797291 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797373 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797401 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797446 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797487 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797596 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797646 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2k7w\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.797839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.797674 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.798079 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.798054 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.798368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.798343 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.798951 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.798822 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.798951 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.798900 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.799106 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.799069 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.801023 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.800999 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.801901 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.801878 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.857615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.857207 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw"] Nov 01 13:00:43.858414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.858384 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2k7w\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.862864 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.862839 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:43.981498 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:43.981467 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:44.398359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.398332 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw"] Nov 01 13:00:44.400113 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:44.400082 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504b8cce_d968_45fc_9b15_508515874fed.slice/crio-5346458be1d9a53aef59d4a075123763a1d01115cb21edb469bf9b02f3f33be3 WatchSource:0}: Error finding container 5346458be1d9a53aef59d4a075123763a1d01115cb21edb469bf9b02f3f33be3: Status 404 returned error can't find the container with id 5346458be1d9a53aef59d4a075123763a1d01115cb21edb469bf9b02f3f33be3 Nov 01 13:00:44.402677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.402640 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:44.402747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.402712 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:44.402747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.402742 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:44.539233 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.539197 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" event={"ID":"504b8cce-d968-45fc-9b15-508515874fed","Type":"ContainerStarted","Data":"6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9"} Nov 01 13:00:44.539447 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.539253 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" event={"ID":"504b8cce-d968-45fc-9b15-508515874fed","Type":"ContainerStarted","Data":"5346458be1d9a53aef59d4a075123763a1d01115cb21edb469bf9b02f3f33be3"} Nov 01 13:00:44.651318 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.651187 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podStartSLOduration=1.6511649400000001 podStartE2EDuration="1.65116494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:44.647971154 +0000 UTC m=+2314.397914131" watchObservedRunningTime="2026-11-01 13:00:44.65116494 +0000 UTC m=+2314.401107923" Nov 01 13:00:44.981759 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:44.981731 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:45.984504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:45.984469 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 01 13:00:45.984898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:45.984524 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 01 13:00:46.559325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:46.541331 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:00:46.982828 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:46.982794 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 01 13:00:46.982987 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:46.982847 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 01 13:00:47.012002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:47.011969 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:47.982836 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:47.982802 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 01 13:00:47.983036 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:47.982860 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 01 13:00:48.982661 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:48.982607 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 01 13:00:48.983194 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:48.982690 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 01 13:00:49.376159 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:49.376039 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:49.496771 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:49.496697 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:49.982149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:49.982108 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 01 13:00:49.982390 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:49.982205 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 01 13:00:50.981920 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:50.981884 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 01 13:00:50.982351 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:50.981953 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 01 13:00:52.264554 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:52.263940 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:52.265783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:52.265760 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:52.283176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:52.283153 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:00:52.391024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:52.390994 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:52.580221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:52.580128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:52.706562 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:52.706415 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:58.398660 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.398625 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:00:58.772805 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.772775 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94"] Nov 01 13:00:58.778045 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.778023 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.867057 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.867018 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94"] Nov 01 13:00:58.908792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.908760 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.908930 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.908836 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bfb\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.908978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.908945 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.909014 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.908997 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.909050 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.909027 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.909095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.909059 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.909222 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.909200 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.909283 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.909266 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:58.909381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:58.909360 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010439 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010411 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010469 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010513 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010561 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010608 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010692 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bfb\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010733 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010772 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010801 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.010921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010876 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.011022 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010916 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.011022 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.010919 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.011179 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.011161 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.011678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.011659 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.012928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.012899 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.013342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.013323 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.063652 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.063604 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.063974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.063953 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bfb\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.092341 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.092288 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:00:59.511847 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.511822 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94"] Nov 01 13:00:59.513716 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:00:59.513682 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b411b1e_c2ba_46fb_8355_d265c9ac6509.slice/crio-e73de69a580809b3f6fd68ba53b7b6cbcb6a330f53f1a03db4a32cf049cad063 WatchSource:0}: Error finding container e73de69a580809b3f6fd68ba53b7b6cbcb6a330f53f1a03db4a32cf049cad063: Status 404 returned error can't find the container with id e73de69a580809b3f6fd68ba53b7b6cbcb6a330f53f1a03db4a32cf049cad063 Nov 01 13:00:59.516157 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.516124 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:59.516231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.516196 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:59.516231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.516227 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:00:59.748972 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.748929 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" event={"ID":"1b411b1e-c2ba-46fb-8355-d265c9ac6509","Type":"ContainerStarted","Data":"35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb"} Nov 01 13:00:59.749164 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.748990 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" event={"ID":"1b411b1e-c2ba-46fb-8355-d265c9ac6509","Type":"ContainerStarted","Data":"e73de69a580809b3f6fd68ba53b7b6cbcb6a330f53f1a03db4a32cf049cad063"} Nov 01 13:00:59.859538 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:00:59.859433 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podStartSLOduration=1.859418315 podStartE2EDuration="1.859418315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:59.856227107 +0000 UTC m=+2329.606170085" watchObservedRunningTime="2026-11-01 13:00:59.859418315 +0000 UTC m=+2329.609361292" Nov 01 13:01:00.093504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:00.093476 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:01:00.094969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:00.094942 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 13:01:00.095088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:00.094988 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 13:01:00.273864 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:00.273823 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:01:01.092607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:01.092575 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 13:01:01.092978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:01.092627 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 13:01:02.092756 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:02.092721 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 13:01:02.093165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:02.092773 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 13:01:02.967664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:02.967623 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv"] Nov 01 13:01:02.973506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:02.973479 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.023496 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.023464 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv"] Nov 01 13:01:03.066646 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.066610 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:01:03.066898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.066872 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:01:03.078507 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.078472 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.078831 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.078800 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.078923 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.078849 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.092969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.092943 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 13:01:03.093321 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.093004 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 13:01:03.180362 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.180282 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.180517 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.180386 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.180517 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.180511 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.181131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.181109 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.182659 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.182639 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.233536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.233512 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p\") pod \"maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.287106 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.287079 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:03.670411 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:01:03.670378 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1eb6c2a_0317_4d5d_bca9_3c964336a4fa.slice/crio-2a771c5632101ad50efd8155ee9ddf29d6bf7fe2d488e9e0dc35ab73d9295e42 WatchSource:0}: Error finding container 2a771c5632101ad50efd8155ee9ddf29d6bf7fe2d488e9e0dc35ab73d9295e42: Status 404 returned error can't find the container with id 2a771c5632101ad50efd8155ee9ddf29d6bf7fe2d488e9e0dc35ab73d9295e42 Nov 01 13:01:03.671221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.671189 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv"] Nov 01 13:01:03.767346 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.767281 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:03.796901 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:03.796869 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" event={"ID":"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa","Type":"ContainerStarted","Data":"2a771c5632101ad50efd8155ee9ddf29d6bf7fe2d488e9e0dc35ab73d9295e42"} Nov 01 13:01:04.107597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:04.105861 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 13:01:04.107597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:04.105927 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 13:01:04.683037 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:04.683004 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:04.807226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:04.807186 4484 generic.go:356] "Generic (PLEG): container finished" podID="b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" containerID="4f77ca7c4651cd07b279e2294679551fe35a44f353db9a4f53e6df481ce95e20" exitCode=0 Nov 01 13:01:04.807468 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:04.807441 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" event={"ID":"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa","Type":"ContainerDied","Data":"4f77ca7c4651cd07b279e2294679551fe35a44f353db9a4f53e6df481ce95e20"} Nov 01 13:01:05.093102 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:05.093068 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 13:01:05.093265 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:05.093132 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 13:01:05.182782 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:05.182752 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:05.626353 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:05.626293 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:05.939760 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:05.939638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:05.975751 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:05.975729 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:06.029323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.026193 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p\") pod \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " Nov 01 13:01:06.029323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.026376 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca\") pod \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " Nov 01 13:01:06.029323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.026416 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca\") pod \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\" (UID: \"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa\") " Nov 01 13:01:06.034451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.034396 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca" pod "b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" (UID: "b1eb6c2a-0317-4d5d-bca9-3c964336a4fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:06.038841 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.038810 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p" pod "b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" (UID: "b1eb6c2a-0317-4d5d-bca9-3c964336a4fa"). InnerVolumeSpecName "kube-api-access-4nd8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:06.044815 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.044788 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca" pod "b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" (UID: "b1eb6c2a-0317-4d5d-bca9-3c964336a4fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:01:06.098118 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.098091 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:01:06.098594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.098570 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:01:06.099452 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.099280 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:01:06.127976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.127947 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-kube-api-access-4nd8p\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:06.127976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.127975 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:06.128146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.127994 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:06.545677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.545642 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:06.822782 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.822749 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" Nov 01 13:01:06.822971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.822748 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv" event={"ID":"b1eb6c2a-0317-4d5d-bca9-3c964336a4fa","Type":"ContainerDied","Data":"2a771c5632101ad50efd8155ee9ddf29d6bf7fe2d488e9e0dc35ab73d9295e42"} Nov 01 13:01:06.822971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:06.822886 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a771c5632101ad50efd8155ee9ddf29d6bf7fe2d488e9e0dc35ab73d9295e42" Nov 01 13:01:07.166801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:07.166722 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:07.460785 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:07.460703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:07.593813 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:07.593766 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:07.731018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:07.730987 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87"] Nov 01 13:01:07.776073 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:07.776038 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b1bd4-zlc87"] Nov 01 13:01:08.090029 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.089950 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:08.368602 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.368530 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7440900-b982-4cba-b878-46f3baf28baf" path="/var/lib/kubelet/pods/d7440900-b982-4cba-b878-46f3baf28baf/volumes" Nov 01 13:01:08.839266 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.839231 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p"] Nov 01 13:01:08.839912 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.839894 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" containerName="revoke-keys" Nov 01 13:01:08.839958 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.839914 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" containerName="revoke-keys" Nov 01 13:01:08.840016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.840006 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" containerName="revoke-keys" Nov 01 13:01:08.843508 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.843489 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.858848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.858822 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83638274-23ff-46a4-bcda-875759a4b92a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.858967 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.858870 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83638274-23ff-46a4-bcda-875759a4b92a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.858967 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.858927 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.858967 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.858951 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.859116 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.858975 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.859116 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.859003 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcglc\" (UniqueName: \"kubernetes.io/projected/83638274-23ff-46a4-bcda-875759a4b92a-kube-api-access-gcglc\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.859182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.859110 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.859182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.859149 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83638274-23ff-46a4-bcda-875759a4b92a-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.859182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.859166 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.930340 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.930296 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p"] Nov 01 13:01:08.960795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.960758 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83638274-23ff-46a4-bcda-875759a4b92a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.960979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.960841 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83638274-23ff-46a4-bcda-875759a4b92a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.960979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.960884 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.960979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.960929 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.960979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.960970 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961097 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcglc\" (UniqueName: \"kubernetes.io/projected/83638274-23ff-46a4-bcda-875759a4b92a-kube-api-access-gcglc\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961154 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961200 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83638274-23ff-46a4-bcda-875759a4b92a-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961226 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961399 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961597 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83638274-23ff-46a4-bcda-875759a4b92a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961705 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.961839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.961822 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.963696 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.963667 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83638274-23ff-46a4-bcda-875759a4b92a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:08.963822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:08.963721 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83638274-23ff-46a4-bcda-875759a4b92a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:09.019917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.019888 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83638274-23ff-46a4-bcda-875759a4b92a-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:09.020040 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.019919 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcglc\" (UniqueName: \"kubernetes.io/projected/83638274-23ff-46a4-bcda-875759a4b92a-kube-api-access-gcglc\") pod \"maas-default-gateway-openshift-default-7df854bf67-8lq5p\" (UID: \"83638274-23ff-46a4-bcda-875759a4b92a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:09.157405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.157322 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:09.576387 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.576361 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p"] Nov 01 13:01:09.578361 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:01:09.578330 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83638274_23ff_46a4_bcda_875759a4b92a.slice/crio-d7fe01691d9be1143d13d6e75fb984dbc9d2b36a644375f46ae6685dc959550e WatchSource:0}: Error finding container d7fe01691d9be1143d13d6e75fb984dbc9d2b36a644375f46ae6685dc959550e: Status 404 returned error can't find the container with id d7fe01691d9be1143d13d6e75fb984dbc9d2b36a644375f46ae6685dc959550e Nov 01 13:01:09.580923 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.580891 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:01:09.580992 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.580955 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:01:09.580992 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.580985 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:01:09.847778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.847693 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" event={"ID":"83638274-23ff-46a4-bcda-875759a4b92a","Type":"ContainerStarted","Data":"80979f4d97f8bbcfbbff73d49a13b72ec34e934ee1fb8677b2aebe3db40f1edf"} Nov 01 13:01:09.847778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.847752 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" event={"ID":"83638274-23ff-46a4-bcda-875759a4b92a","Type":"ContainerStarted","Data":"d7fe01691d9be1143d13d6e75fb984dbc9d2b36a644375f46ae6685dc959550e"} Nov 01 13:01:09.960384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:09.960331 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podStartSLOduration=1.960286561 podStartE2EDuration="1.960286561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:09.956939428 +0000 UTC m=+2339.706882407" watchObservedRunningTime="2026-11-01 13:01:09.960286561 +0000 UTC m=+2339.710229539" Nov 01 13:01:10.164384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:10.163641 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:10.166358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:10.165111 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:10.166358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:10.165175 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:11.161325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:11.160718 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:11.161325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:11.160784 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:11.605178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:11.605145 4484 scope.go:122] "RemoveContainer" containerID="581b7380508df298f220e12780104170369681e5671e7f6a6d41e84ad876eb60" Nov 01 13:01:12.159724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.159681 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:12.160049 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.160016 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:12.457883 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.457764 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:12.592059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.592023 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:12.812483 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.812443 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:12.818830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.818800 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq"] Nov 01 13:01:12.819034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:12.819012 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" containerID="cri-o://fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b" gracePeriod=30 Nov 01 13:01:13.160381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.157671 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:13.160381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.157732 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:13.202657 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.202616 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:13.329976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.329940 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:13.531231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.531193 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:13.866380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.865773 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6"] Nov 01 13:01:13.872828 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.872801 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:13.907686 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.907657 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6"] Nov 01 13:01:13.932408 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.932372 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:13.932594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.932429 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzljk\" (UniqueName: \"kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:13.932594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.932516 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:13.970325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.965745 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:01:13.970325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:13.966609 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:01:14.033460 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.033427 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.033460 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.033468 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzljk\" (UniqueName: \"kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.033682 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.033495 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.034133 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.034110 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.035691 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.035665 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.086484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.086455 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzljk\" (UniqueName: \"kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk\") pod \"maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.163377 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.162750 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:14.163377 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.162814 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:14.186717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.186682 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:14.571893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.571863 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6"] Nov 01 13:01:14.573751 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:01:14.573713 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95871eb_cad9_46da_83e4_7a212982d679.slice/crio-ee1735a5d4a312bc1b2144e168e1b937b381db3242077c915888cfed410185a9 WatchSource:0}: Error finding container ee1735a5d4a312bc1b2144e168e1b937b381db3242077c915888cfed410185a9: Status 404 returned error can't find the container with id ee1735a5d4a312bc1b2144e168e1b937b381db3242077c915888cfed410185a9 Nov 01 13:01:14.589178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.589155 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:14.724287 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.724257 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:14.889070 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.888987 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq"] Nov 01 13:01:14.890068 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.890035 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" containerID="cri-o://d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487" gracePeriod=30 Nov 01 13:01:14.906766 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.906732 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" event={"ID":"d95871eb-cad9-46da-83e4-7a212982d679","Type":"ContainerStarted","Data":"ee1735a5d4a312bc1b2144e168e1b937b381db3242077c915888cfed410185a9"} Nov 01 13:01:14.919256 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:14.919230 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:15.164380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:15.161996 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:15.164380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:15.162054 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:15.835843 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:15.835809 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x"] Nov 01 13:01:15.917034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:15.916939 4484 generic.go:356] "Generic (PLEG): container finished" podID="d95871eb-cad9-46da-83e4-7a212982d679" containerID="46bd0bcaeb1f13430027d75e9dfdbbf5a4bb91736b3a23a05fcbf65504445a6b" exitCode=0 Nov 01 13:01:15.917190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:15.917028 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" event={"ID":"d95871eb-cad9-46da-83e4-7a212982d679","Type":"ContainerDied","Data":"46bd0bcaeb1f13430027d75e9dfdbbf5a4bb91736b3a23a05fcbf65504445a6b"} Nov 01 13:01:15.939281 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:15.939251 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw"] Nov 01 13:01:16.158496 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:16.158464 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:16.158688 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:16.158524 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:16.203507 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:16.203424 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:16.329499 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:16.329469 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:17.082418 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.082390 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:17.176422 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.172445 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 13:01:17.176422 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.172517 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 13:01:17.210986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.210946 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca\") pod \"d95871eb-cad9-46da-83e4-7a212982d679\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " Nov 01 13:01:17.211261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.211224 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca\") pod \"d95871eb-cad9-46da-83e4-7a212982d679\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " Nov 01 13:01:17.211445 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.211359 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk\" (UniqueName: \"kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk\") pod \"d95871eb-cad9-46da-83e4-7a212982d679\" (UID: \"d95871eb-cad9-46da-83e4-7a212982d679\") " Nov 01 13:01:17.222910 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.222870 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk" pod "d95871eb-cad9-46da-83e4-7a212982d679" (UID: "d95871eb-cad9-46da-83e4-7a212982d679"). InnerVolumeSpecName "kube-api-access-zzljk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:17.235608 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.235573 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca" pod "d95871eb-cad9-46da-83e4-7a212982d679" (UID: "d95871eb-cad9-46da-83e4-7a212982d679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:17.248546 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.237376 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca" pod "d95871eb-cad9-46da-83e4-7a212982d679" (UID: "d95871eb-cad9-46da-83e4-7a212982d679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:01:17.325327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.321422 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:17.325327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.321464 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95871eb-cad9-46da-83e4-7a212982d679-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:17.325327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.321482 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzljk\" (UniqueName: \"kubernetes.io/projected/d95871eb-cad9-46da-83e4-7a212982d679-kube-api-access-zzljk\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:17.933282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.933246 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" event={"ID":"d95871eb-cad9-46da-83e4-7a212982d679","Type":"ContainerDied","Data":"ee1735a5d4a312bc1b2144e168e1b937b381db3242077c915888cfed410185a9"} Nov 01 13:01:17.933510 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.933368 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1735a5d4a312bc1b2144e168e1b937b381db3242077c915888cfed410185a9" Nov 01 13:01:17.933510 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:17.933419 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6" Nov 01 13:01:18.234759 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.234731 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 13:01:18.280859 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.280824 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n"] Nov 01 13:01:18.340132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340101 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340156 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340196 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340221 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340283 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340378 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340410 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340468 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340523 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket\") pod \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\" (UID: \"4fc4d29d-9be9-42e5-a95a-59df4aa746f0\") " Nov 01 13:01:18.340576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340542 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:18.340846 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340726 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:18.340949 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340926 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:18.341160 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.341022 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.341244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.341177 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.341244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.340963 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:18.341244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.341170 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:18.368531 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.366435 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:18.368531 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.366759 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:18.383573 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.379797 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:01:18.384392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.384345 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq" pod "4fc4d29d-9be9-42e5-a95a-59df4aa746f0" (UID: "4fc4d29d-9be9-42e5-a95a-59df4aa746f0"). InnerVolumeSpecName "kube-api-access-rd2dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:18.411973 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.411949 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472011 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472044 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2dq\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-kube-api-access-rd2dq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472060 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472074 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472091 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472104 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.472226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.472118 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc4d29d-9be9-42e5-a95a-59df4aa746f0-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:18.687221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.687188 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:18.687522 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.687496 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:18.688383 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.688363 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" Nov 01 13:01:18.962325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.960975 4484 generic.go:356] "Generic (PLEG): container finished" podID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerID="fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b" exitCode=0 Nov 01 13:01:18.962325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.961950 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" Nov 01 13:01:18.975330 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.973392 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" event={"ID":"4fc4d29d-9be9-42e5-a95a-59df4aa746f0","Type":"ContainerDied","Data":"fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b"} Nov 01 13:01:18.975330 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.973463 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq" event={"ID":"4fc4d29d-9be9-42e5-a95a-59df4aa746f0","Type":"ContainerDied","Data":"cf04e0a494cbcb87fb07cfe6db391f0e6696e5031caa8ae9c09f08b7b21b7dbd"} Nov 01 13:01:18.975330 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.973495 4484 scope.go:122] "RemoveContainer" containerID="fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b" Nov 01 13:01:18.986813 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.986779 4484 scope.go:122] "RemoveContainer" containerID="2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7" Nov 01 13:01:18.996802 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.996782 4484 scope.go:122] "RemoveContainer" containerID="fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b" Nov 01 13:01:18.997113 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:18.997081 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b\": container with ID starting with fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b not found: ID does not exist" containerID="fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b" Nov 01 13:01:18.997165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.997123 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b"} err="failed to get container status \"fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b\": rpc error: code = NotFound desc = could not find container \"fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b\": container with ID starting with fa13025f53ec94b88215033533b73e00c2a2e046a0ebd8d7814f21d2da9b233b not found: ID does not exist" Nov 01 13:01:18.997165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.997140 4484 scope.go:122] "RemoveContainer" containerID="2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7" Nov 01 13:01:18.997457 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:18.997436 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7\": container with ID starting with 2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7 not found: ID does not exist" containerID="2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7" Nov 01 13:01:18.997543 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:18.997464 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7"} err="failed to get container status \"2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7\": rpc error: code = NotFound desc = could not find container \"2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7\": container with ID starting with 2414b86616a366ec5c165d38bdd048c4b9fa0c5fdf8e24306398c30f52ba86f7 not found: ID does not exist" Nov 01 13:01:19.128561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:19.128528 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq"] Nov 01 13:01:19.131743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:19.131716 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2810c2-openshift-default-5fb776b75f-dhbhq"] Nov 01 13:01:19.295700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:19.295669 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:20.184605 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.184582 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 13:01:20.300177 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300143 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300186 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300244 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300377 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300444 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300447 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300524 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300584 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.300687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300681 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo\") pod \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\" (UID: \"8c195962-b11f-4f79-ae40-96ca0f1bd60e\") " Nov 01 13:01:20.301164 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300699 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:20.301164 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.300944 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:20.301282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.301236 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.301282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.301258 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.301282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.301273 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.301459 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.301432 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:20.301597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.301489 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:20.303188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.303149 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "kube-api-access-t4flp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:20.303334 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.303276 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:20.303652 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.303633 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:01:20.304275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.304256 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token" pod "8c195962-b11f-4f79-ae40-96ca0f1bd60e" (UID: "8c195962-b11f-4f79-ae40-96ca0f1bd60e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:20.377690 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.377653 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" path="/var/lib/kubelet/pods/4fc4d29d-9be9-42e5-a95a-59df4aa746f0/volumes" Nov 01 13:01:20.402635 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.402614 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.402734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.402639 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.402734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.402650 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.402734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.402659 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c195962-b11f-4f79-ae40-96ca0f1bd60e-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.402734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.402667 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c195962-b11f-4f79-ae40-96ca0f1bd60e-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.402734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.402677 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4flp\" (UniqueName: \"kubernetes.io/projected/8c195962-b11f-4f79-ae40-96ca0f1bd60e-kube-api-access-t4flp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:20.836871 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.836809 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" podUID="e5e70166-b969-4d3d-baa1-dae474808bf3" containerName="payload-processing" containerID="cri-o://6ce0fa0964ad702450f49f898072fc910b6460e726416bd21323af7982a6c02c" gracePeriod=25 Nov 01 13:01:20.940037 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.939992 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" podUID="c66ddcd7-a9fd-4d00-8aef-9da273265a62" containerName="payload-pre-processing" containerID="cri-o://dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a" gracePeriod=25 Nov 01 13:01:20.981129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.981090 4484 generic.go:356] "Generic (PLEG): container finished" podID="e5e70166-b969-4d3d-baa1-dae474808bf3" containerID="6ce0fa0964ad702450f49f898072fc910b6460e726416bd21323af7982a6c02c" exitCode=0 Nov 01 13:01:20.981244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.981156 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" event={"ID":"e5e70166-b969-4d3d-baa1-dae474808bf3","Type":"ContainerDied","Data":"6ce0fa0964ad702450f49f898072fc910b6460e726416bd21323af7982a6c02c"} Nov 01 13:01:20.982780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.982758 4484 generic.go:356] "Generic (PLEG): container finished" podID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerID="d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487" exitCode=0 Nov 01 13:01:20.982880 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.982807 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" Nov 01 13:01:20.982938 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.982809 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" event={"ID":"8c195962-b11f-4f79-ae40-96ca0f1bd60e","Type":"ContainerDied","Data":"d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487"} Nov 01 13:01:20.982938 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.982926 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq" event={"ID":"8c195962-b11f-4f79-ae40-96ca0f1bd60e","Type":"ContainerDied","Data":"c06cf21bbae7fdb4c027e2d3538acdf83cc0ce3e89a1da9436293a504551a976"} Nov 01 13:01:20.983048 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:20.982958 4484 scope.go:122] "RemoveContainer" containerID="d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487" Nov 01 13:01:21.087155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.087091 4484 scope.go:122] "RemoveContainer" containerID="d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487" Nov 01 13:01:21.087528 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:21.087502 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487\": container with ID starting with d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487 not found: ID does not exist" containerID="d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487" Nov 01 13:01:21.087622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.087557 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487"} err="failed to get container status \"d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487\": rpc error: code = NotFound desc = could not find container \"d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487\": container with ID starting with d33fdf7c3c23a54c5d5f5ad995e257c937f65432a5deb095b665eab400f09487 not found: ID does not exist" Nov 01 13:01:21.139179 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.139153 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 13:01:21.185867 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.185824 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq"] Nov 01 13:01:21.211256 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.211222 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume\") pod \"e5e70166-b969-4d3d-baa1-dae474808bf3\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " Nov 01 13:01:21.220207 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.211427 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp\" (UniqueName: \"kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp\") pod \"e5e70166-b969-4d3d-baa1-dae474808bf3\" (UID: \"e5e70166-b969-4d3d-baa1-dae474808bf3\") " Nov 01 13:01:21.220207 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.211581 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume" pod "e5e70166-b969-4d3d-baa1-dae474808bf3" (UID: "e5e70166-b969-4d3d-baa1-dae474808bf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:21.220207 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.211855 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e70166-b969-4d3d-baa1-dae474808bf3-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:21.220207 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.214157 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp" pod "e5e70166-b969-4d3d-baa1-dae474808bf3" (UID: "e5e70166-b969-4d3d-baa1-dae474808bf3"). InnerVolumeSpecName "kube-api-access-4fwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:21.231874 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.231847 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-phqhq"] Nov 01 13:01:21.277612 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.277584 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 13:01:21.312721 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.312684 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwbp\" (UniqueName: \"kubernetes.io/projected/e5e70166-b969-4d3d-baa1-dae474808bf3-kube-api-access-4fwbp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:21.413796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.413713 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc\" (UniqueName: \"kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc\") pod \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " Nov 01 13:01:21.413796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.413750 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume\") pod \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\" (UID: \"c66ddcd7-a9fd-4d00-8aef-9da273265a62\") " Nov 01 13:01:21.414215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.414192 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume" pod "c66ddcd7-a9fd-4d00-8aef-9da273265a62" (UID: "c66ddcd7-a9fd-4d00-8aef-9da273265a62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:21.414358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.414342 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c66ddcd7-a9fd-4d00-8aef-9da273265a62-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:21.415778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.415758 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc" pod "c66ddcd7-a9fd-4d00-8aef-9da273265a62" (UID: "c66ddcd7-a9fd-4d00-8aef-9da273265a62"). InnerVolumeSpecName "kube-api-access-l9zxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:21.515819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.515780 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zxc\" (UniqueName: \"kubernetes.io/projected/c66ddcd7-a9fd-4d00-8aef-9da273265a62-kube-api-access-l9zxc\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:21.994096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.994063 4484 generic.go:356] "Generic (PLEG): container finished" podID="c66ddcd7-a9fd-4d00-8aef-9da273265a62" containerID="dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a" exitCode=0 Nov 01 13:01:21.994261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.994123 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" Nov 01 13:01:21.994261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.994145 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" event={"ID":"c66ddcd7-a9fd-4d00-8aef-9da273265a62","Type":"ContainerDied","Data":"dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a"} Nov 01 13:01:21.994261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.994196 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw" event={"ID":"c66ddcd7-a9fd-4d00-8aef-9da273265a62","Type":"ContainerDied","Data":"4b4f2bb8723738ceacb49485521e184e5da9433eca0e55d15b177d51a6859463"} Nov 01 13:01:21.994261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.994227 4484 scope.go:122] "RemoveContainer" containerID="dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a" Nov 01 13:01:21.995579 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.995563 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" Nov 01 13:01:21.995670 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:21.995582 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x" event={"ID":"e5e70166-b969-4d3d-baa1-dae474808bf3","Type":"ContainerDied","Data":"3801ac8e46bfc9d618f5738f3bf46a5feaad32e59bbbc737ac54fb3f46114679"} Nov 01 13:01:22.005650 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.005527 4484 scope.go:122] "RemoveContainer" containerID="dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a" Nov 01 13:01:22.005844 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:22.005818 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a\": container with ID starting with dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a not found: ID does not exist" containerID="dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a" Nov 01 13:01:22.005939 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.005855 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a"} err="failed to get container status \"dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a\": rpc error: code = NotFound desc = could not find container \"dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a\": container with ID starting with dafc7f681e1c0569f1ba40d0b915f3965d9c09036a848da6f21f479d39670c8a not found: ID does not exist" Nov 01 13:01:22.005939 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.005880 4484 scope.go:122] "RemoveContainer" containerID="6ce0fa0964ad702450f49f898072fc910b6460e726416bd21323af7982a6c02c" Nov 01 13:01:22.171935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.171834 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw"] Nov 01 13:01:22.217761 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.217728 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-czmdw"] Nov 01 13:01:22.279350 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.279287 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x"] Nov 01 13:01:22.283475 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.283447 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-6hp8x"] Nov 01 13:01:22.372869 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.372838 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" path="/var/lib/kubelet/pods/8c195962-b11f-4f79-ae40-96ca0f1bd60e/volumes" Nov 01 13:01:22.373260 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.373249 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66ddcd7-a9fd-4d00-8aef-9da273265a62" path="/var/lib/kubelet/pods/c66ddcd7-a9fd-4d00-8aef-9da273265a62/volumes" Nov 01 13:01:22.373645 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:22.373631 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e70166-b969-4d3d-baa1-dae474808bf3" path="/var/lib/kubelet/pods/e5e70166-b969-4d3d-baa1-dae474808bf3/volumes" Nov 01 13:01:24.706753 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:24.706714 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:25.701511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:25.701481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:25.828928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:25.828787 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:27.044490 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.044451 4484 generic.go:356] "Generic (PLEG): container finished" podID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerID="eb6b4c103648e7bd4ccecc67e8340243ef2510d321cafc135a1d41250c6e6e29" exitCode=0 Nov 01 13:01:27.045351 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.045315 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" event={"ID":"e8443af5-fdb8-4072-9e1b-567609af5e6e","Type":"ContainerDied","Data":"eb6b4c103648e7bd4ccecc67e8340243ef2510d321cafc135a1d41250c6e6e29"} Nov 01 13:01:27.046149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.046127 4484 scope.go:122] "RemoveContainer" containerID="eb6b4c103648e7bd4ccecc67e8340243ef2510d321cafc135a1d41250c6e6e29" Nov 01 13:01:27.048849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.048812 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:01:27.049074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.049049 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:01:27.049221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.049204 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:01:27.720481 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:27.720445 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:01:28.072589 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:28.072550 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" event={"ID":"e8443af5-fdb8-4072-9e1b-567609af5e6e","Type":"ContainerStarted","Data":"b035f905568a2042ae25d60eff2b6f4858cae9bbd475ed4e8ff0e125f2c218b5"} Nov 01 13:01:28.106941 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:28.106901 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:01:28.108262 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:28.108235 4484 patch_prober.go:29] interesting pod/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 13:01:28.108412 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:28.108282 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 13:01:29.106799 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.106766 4484 patch_prober.go:29] interesting pod/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 13:01:29.107222 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.106828 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 13:01:29.467700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.467612 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8"] Nov 01 13:01:29.468437 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468416 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" Nov 01 13:01:29.468505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468439 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" Nov 01 13:01:29.468505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468470 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c66ddcd7-a9fd-4d00-8aef-9da273265a62" containerName="payload-pre-processing" Nov 01 13:01:29.468505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468479 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66ddcd7-a9fd-4d00-8aef-9da273265a62" containerName="payload-pre-processing" Nov 01 13:01:29.468505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468498 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468506 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468530 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d95871eb-cad9-46da-83e4-7a212982d679" containerName="revoke-keys" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468537 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95871eb-cad9-46da-83e4-7a212982d679" containerName="revoke-keys" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468567 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468574 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468584 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e5e70166-b969-4d3d-baa1-dae474808bf3" containerName="payload-processing" Nov 01 13:01:29.468640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468590 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e70166-b969-4d3d-baa1-dae474808bf3" containerName="payload-processing" Nov 01 13:01:29.468856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468728 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" Nov 01 13:01:29.468856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468743 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e70166-b969-4d3d-baa1-dae474808bf3" containerName="payload-processing" Nov 01 13:01:29.468856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468755 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c195962-b11f-4f79-ae40-96ca0f1bd60e" containerName="istio-proxy" Nov 01 13:01:29.468856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468767 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66ddcd7-a9fd-4d00-8aef-9da273265a62" containerName="payload-pre-processing" Nov 01 13:01:29.468856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468777 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc4d29d-9be9-42e5-a95a-59df4aa746f0" containerName="istio-proxy" Nov 01 13:01:29.468856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.468788 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95871eb-cad9-46da-83e4-7a212982d679" containerName="revoke-keys" Nov 01 13:01:29.472752 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.472736 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.482956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.482936 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.483055 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.482974 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwc8\" (UniqueName: \"kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.483055 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.483005 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.525400 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.525364 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8"] Nov 01 13:01:29.568564 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.568535 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:01:29.568877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.568850 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:01:29.584653 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.584622 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.584866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.584845 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.584939 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.584904 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwc8\" (UniqueName: \"kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.585470 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.585445 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.587613 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.587593 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.639544 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.639502 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwc8\" (UniqueName: \"kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8\") pod \"maas-api-revoke-sub-dca3d77b-6wgk8\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:29.681339 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.681293 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:29.786349 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:29.786312 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:30.107975 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:30.107880 4484 patch_prober.go:29] interesting pod/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 13:01:30.107975 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:30.107951 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 13:01:30.222342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:30.222314 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8"] Nov 01 13:01:30.223826 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:01:30.223795 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9ada65_8080_43ed_aadb_d5fc97e024b5.slice/crio-8ed751dd1477f36f3d84ee4eaafbb1248d8a16c97518f0ffcc968c2c17049643 WatchSource:0}: Error finding container 8ed751dd1477f36f3d84ee4eaafbb1248d8a16c97518f0ffcc968c2c17049643: Status 404 returned error can't find the container with id 8ed751dd1477f36f3d84ee4eaafbb1248d8a16c97518f0ffcc968c2c17049643 Nov 01 13:01:30.369379 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:30.366477 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:01:31.106625 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.106529 4484 patch_prober.go:29] interesting pod/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 13:01:31.106842 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.106820 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 13:01:31.110634 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.110610 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" event={"ID":"3c9ada65-8080-43ed-aadb-d5fc97e024b5","Type":"ContainerStarted","Data":"aacd41d5f97a0e04460d356a355b957811e930438802144a772589b9905c14da"} Nov 01 13:01:31.110944 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.110658 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" event={"ID":"3c9ada65-8080-43ed-aadb-d5fc97e024b5","Type":"ContainerStarted","Data":"8ed751dd1477f36f3d84ee4eaafbb1248d8a16c97518f0ffcc968c2c17049643"} Nov 01 13:01:31.225522 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.225470 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" podStartSLOduration=2.225453521 podStartE2EDuration="2.225453521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:31.222648777 +0000 UTC m=+2360.972591755" watchObservedRunningTime="2026-11-01 13:01:31.225453521 +0000 UTC m=+2360.975396498" Nov 01 13:01:31.461454 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.441528 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:31.687069 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.687017 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:31.808564 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:31.808530 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:32.017979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.017941 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv"] Nov 01 13:01:32.067253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.067154 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187d06ed-pbdzv"] Nov 01 13:01:32.106448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.106416 4484 patch_prober.go:29] interesting pod/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 13:01:32.106614 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.106466 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 13:01:32.118791 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.118760 4484 generic.go:356] "Generic (PLEG): container finished" podID="3c9ada65-8080-43ed-aadb-d5fc97e024b5" containerID="aacd41d5f97a0e04460d356a355b957811e930438802144a772589b9905c14da" exitCode=0 Nov 01 13:01:32.119215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.118990 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" event={"ID":"3c9ada65-8080-43ed-aadb-d5fc97e024b5","Type":"ContainerDied","Data":"aacd41d5f97a0e04460d356a355b957811e930438802144a772589b9905c14da"} Nov 01 13:01:32.372319 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.372226 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1eb6c2a-0317-4d5d-bca9-3c964336a4fa" path="/var/lib/kubelet/pods/b1eb6c2a-0317-4d5d-bca9-3c964336a4fa/volumes" Nov 01 13:01:32.403217 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:32.403179 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:33.107121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.107079 4484 patch_prober.go:29] interesting pod/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 13:01:33.107336 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.107157 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 13:01:33.269316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.269285 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:33.304914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.304876 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" containerID="cri-o://cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221" gracePeriod=45 Nov 01 13:01:33.370792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.370707 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca\") pod \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " Nov 01 13:01:33.370944 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.370818 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca\") pod \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " Nov 01 13:01:33.370944 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.370859 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8\" (UniqueName: \"kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8\") pod \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\" (UID: \"3c9ada65-8080-43ed-aadb-d5fc97e024b5\") " Nov 01 13:01:33.371226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.371200 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca" pod "3c9ada65-8080-43ed-aadb-d5fc97e024b5" (UID: "3c9ada65-8080-43ed-aadb-d5fc97e024b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:01:33.371294 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.371275 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.372744 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.372716 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca" pod "3c9ada65-8080-43ed-aadb-d5fc97e024b5" (UID: "3c9ada65-8080-43ed-aadb-d5fc97e024b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:01:33.372866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.372815 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8" pod "3c9ada65-8080-43ed-aadb-d5fc97e024b5" (UID: "3c9ada65-8080-43ed-aadb-d5fc97e024b5"). InnerVolumeSpecName "kube-api-access-sqwc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:33.473235 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.473205 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9ada65-8080-43ed-aadb-d5fc97e024b5-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.473235 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.473230 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqwc8\" (UniqueName: \"kubernetes.io/projected/3c9ada65-8080-43ed-aadb-d5fc97e024b5-kube-api-access-sqwc8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.570522 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.570496 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.679501 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir\") pod \"97ed692d-230c-4d02-8475-eec80bab204f\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.679566 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd\" (UniqueName: \"kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd\") pod \"97ed692d-230c-4d02-8475-eec80bab204f\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.679650 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home\") pod \"97ed692d-230c-4d02-8475-eec80bab204f\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.679687 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs\" (UniqueName: \"kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs\") pod \"97ed692d-230c-4d02-8475-eec80bab204f\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.681061 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home" pod "97ed692d-230c-4d02-8475-eec80bab204f" (UID: "97ed692d-230c-4d02-8475-eec80bab204f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.685829 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache\") pod \"97ed692d-230c-4d02-8475-eec80bab204f\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.685883 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm\") pod \"97ed692d-230c-4d02-8475-eec80bab204f\" (UID: \"97ed692d-230c-4d02-8475-eec80bab204f\") " Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.686396 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache" pod "97ed692d-230c-4d02-8475-eec80bab204f" (UID: "97ed692d-230c-4d02-8475-eec80bab204f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.686609 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.686639 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.692391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.690468 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir" pod "97ed692d-230c-4d02-8475-eec80bab204f" (UID: "97ed692d-230c-4d02-8475-eec80bab204f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:33.697819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.696989 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd" pod "97ed692d-230c-4d02-8475-eec80bab204f" (UID: "97ed692d-230c-4d02-8475-eec80bab204f"). InnerVolumeSpecName "kube-api-access-d8bsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:01:33.697819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.697515 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm" pod "97ed692d-230c-4d02-8475-eec80bab204f" (UID: "97ed692d-230c-4d02-8475-eec80bab204f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:01:33.699025 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.698990 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs" pod "97ed692d-230c-4d02-8475-eec80bab204f" (UID: "97ed692d-230c-4d02-8475-eec80bab204f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:01:33.788153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.788113 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ed692d-230c-4d02-8475-eec80bab204f-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.788153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.788151 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.788420 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.788163 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ed692d-230c-4d02-8475-eec80bab204f-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:33.788420 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:33.788178 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bsd\" (UniqueName: \"kubernetes.io/projected/97ed692d-230c-4d02-8475-eec80bab204f-kube-api-access-d8bsd\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:01:34.111222 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.111193 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:01:34.111722 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.111705 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:01:34.112456 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.112438 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:01:34.131263 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.131241 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" Nov 01 13:01:34.131404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.131279 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8" event={"ID":"3c9ada65-8080-43ed-aadb-d5fc97e024b5","Type":"ContainerDied","Data":"8ed751dd1477f36f3d84ee4eaafbb1248d8a16c97518f0ffcc968c2c17049643"} Nov 01 13:01:34.131404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.131337 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ed751dd1477f36f3d84ee4eaafbb1248d8a16c97518f0ffcc968c2c17049643" Nov 01 13:01:34.132810 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.132786 4484 generic.go:356] "Generic (PLEG): container finished" podID="97ed692d-230c-4d02-8475-eec80bab204f" containerID="cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221" exitCode=0 Nov 01 13:01:34.132911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.132839 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" Nov 01 13:01:34.132911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.132869 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" event={"ID":"97ed692d-230c-4d02-8475-eec80bab204f","Type":"ContainerDied","Data":"cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221"} Nov 01 13:01:34.133028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.132911 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" event={"ID":"97ed692d-230c-4d02-8475-eec80bab204f","Type":"ContainerDied","Data":"c977ce32b15f8319913c9deeeb05aa29730a2c2c64e70e8e5586d126ad5460ac"} Nov 01 13:01:34.133028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.132935 4484 scope.go:122] "RemoveContainer" containerID="cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221" Nov 01 13:01:34.143371 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.143352 4484 scope.go:122] "RemoveContainer" containerID="cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221" Nov 01 13:01:34.143664 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:01:34.143645 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221\": container with ID starting with cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221 not found: ID does not exist" containerID="cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221" Nov 01 13:01:34.143724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.143670 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221"} err="failed to get container status \"cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221\": rpc error: code = NotFound desc = could not find container \"cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221\": container with ID starting with cbf6f93bb6a97fc25b981a738102ee059f72c3855dd29c18ba93790dce76f221 not found: ID does not exist" Nov 01 13:01:34.497737 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.497700 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n"] Nov 01 13:01:34.501786 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.501759 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n"] Nov 01 13:01:34.541832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:34.541788 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-2-w5-9514b0-kserve-7f788b95c8-65x5n" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 13:01:36.369567 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:36.369537 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ed692d-230c-4d02-8475-eec80bab204f" path="/var/lib/kubelet/pods/97ed692d-230c-4d02-8475-eec80bab204f/volumes" Nov 01 13:01:37.452665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:37.452631 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:37.568031 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:37.567988 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:41.063021 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:41.062983 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:01:41.482744 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:41.482717 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:43.113795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:43.113765 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:44.524477 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:44.524443 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:48.591394 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:48.591363 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:49.182210 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:49.182184 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:49.307839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:49.307803 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:53.987079 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:53.987005 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:56.166853 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:56.166820 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:56.289215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:56.289184 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:56.432574 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:56.432492 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:01:59.288947 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:01:59.288914 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:00.251315 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:00.251265 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:02:01.919004 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:01.918964 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:03.915914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:03.915876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:05.643917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:05.643881 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:05.899024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:05.898947 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:06.031323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:06.031230 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:06.212368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:06.212258 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6"] Nov 01 13:02:06.258363 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:06.258329 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9514b0-r64t6"] Nov 01 13:02:06.367775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:06.367746 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95871eb-cad9-46da-83e4-7a212982d679" path="/var/lib/kubelet/pods/d95871eb-cad9-46da-83e4-7a212982d679/volumes" Nov 01 13:02:06.593881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:06.593832 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:11.671324 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:11.664807 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:11.704435 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:11.704406 4484 scope.go:122] "RemoveContainer" containerID="65779fa790e1456112b60bf843496ec3fc4a845691c656258016fd9cd4a63b7a" Nov 01 13:02:11.713640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:11.713618 4484 scope.go:122] "RemoveContainer" containerID="6351f6309f47003338abcfce12b51cb5af4444650785e5d2404de4fd475a0595" Nov 01 13:02:12.373329 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:12.373281 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:12.690962 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:12.690879 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9"] Nov 01 13:02:12.691440 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:12.691137 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" containerID="cri-o://7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1" gracePeriod=30 Nov 01 13:02:14.009684 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:14.009626 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:14.258664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:14.258617 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:15.120648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:15.120613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:16.984813 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:16.984767 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:17.954617 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.954562 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 13:02:17.975799 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.975774 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.975931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.975826 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.975931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.975865 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976058 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.975945 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976058 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976002 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976058 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976036 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976054 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:17.976221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976081 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976155 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:17.976221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976177 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976235 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data\") pod \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\" (UID: \"c5a4d823-1015-41d5-87d5-962ea45b4ae7\") " Nov 01 13:02:17.976534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976399 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:17.976534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976429 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:02:17.976714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976618 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:17.976774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976718 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:17.976774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976732 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:17.976774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976743 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:17.976774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976755 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:17.976774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.976766 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:17.978252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.978176 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:17.978388 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.978265 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "kube-api-access-wbwtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:17.978640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.978622 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:02:17.978703 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:17.978627 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy" pod "c5a4d823-1015-41d5-87d5-962ea45b4ae7" (UID: "c5a4d823-1015-41d5-87d5-962ea45b4ae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:18.078448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.078417 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwtz\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-kube-api-access-wbwtz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:18.078448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.078451 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:18.078834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.078466 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:18.078834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.078484 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a4d823-1015-41d5-87d5-962ea45b4ae7-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:18.429724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.429689 4484 generic.go:356] "Generic (PLEG): container finished" podID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerID="7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1" exitCode=0 Nov 01 13:02:18.429884 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.429741 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" Nov 01 13:02:18.429884 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.429773 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" event={"ID":"c5a4d823-1015-41d5-87d5-962ea45b4ae7","Type":"ContainerDied","Data":"7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1"} Nov 01 13:02:18.429884 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.429837 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9" event={"ID":"c5a4d823-1015-41d5-87d5-962ea45b4ae7","Type":"ContainerDied","Data":"e6f505f26a3b3e3aecd1c7137c1367a545ddc4377644b8b14918e4c88b673200"} Nov 01 13:02:18.429884 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.429867 4484 scope.go:122] "RemoveContainer" containerID="7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1" Nov 01 13:02:18.439388 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.439369 4484 scope.go:122] "RemoveContainer" containerID="b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014" Nov 01 13:02:18.448096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.448080 4484 scope.go:122] "RemoveContainer" containerID="7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1" Nov 01 13:02:18.448334 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:18.448295 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1\": container with ID starting with 7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1 not found: ID does not exist" containerID="7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1" Nov 01 13:02:18.448452 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.448338 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1"} err="failed to get container status \"7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1\": rpc error: code = NotFound desc = could not find container \"7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1\": container with ID starting with 7cb6bb3bdcbeb0109cf2217d60c39eb502df497fabbc7f5dc788c6e1013facc1 not found: ID does not exist" Nov 01 13:02:18.448452 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.448354 4484 scope.go:122] "RemoveContainer" containerID="b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014" Nov 01 13:02:18.448614 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:18.448591 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014\": container with ID starting with b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014 not found: ID does not exist" containerID="b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014" Nov 01 13:02:18.448656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.448613 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014"} err="failed to get container status \"b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014\": rpc error: code = NotFound desc = could not find container \"b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014\": container with ID starting with b515981c3f01a0e5f710dead618e52d6742e481c50c1e0c043dd7da244807014 not found: ID does not exist" Nov 01 13:02:18.583726 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.583701 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9"] Nov 01 13:02:18.586643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:18.586622 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-gslz9"] Nov 01 13:02:20.370253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:20.370211 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" path="/var/lib/kubelet/pods/c5a4d823-1015-41d5-87d5-962ea45b4ae7/volumes" Nov 01 13:02:20.538909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:20.538876 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:20.952381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:20.952345 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:22.294042 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:22.294007 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:22.616197 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:22.616108 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:02:27.701983 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:27.701954 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:28.620875 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:28.620845 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:28.758511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:28.758474 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:28.910798 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:28.910711 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:30.253328 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:30.253273 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:02:33.135390 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:33.135327 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:35.447965 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.447916 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:35.667082 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.667050 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:35.821837 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.821791 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:35.822648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.822622 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76"] Nov 01 13:02:35.822892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.822869 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" containerID="cri-o://e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa" gracePeriod=30 Nov 01 13:02:35.841899 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.841874 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw"] Nov 01 13:02:35.842073 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.842054 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" containerID="cri-o://6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9" gracePeriod=30 Nov 01 13:02:35.861098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.861065 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94"] Nov 01 13:02:35.861264 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.861245 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" containerID="cri-o://35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb" gracePeriod=30 Nov 01 13:02:35.867723 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.867700 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh"] Nov 01 13:02:35.867892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.867874 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" containerID="cri-o://15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b" gracePeriod=30 Nov 01 13:02:35.986675 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.986637 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 13:02:35.986844 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:35.986709 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 13:02:36.101260 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:36.101174 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 13:02:36.101260 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:36.101247 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 13:02:36.210953 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:36.210917 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 13:02:36.211135 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:36.210968 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 13:02:37.268320 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:37.268275 4484 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 13:02:37.268749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:37.268347 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 13:02:38.424750 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:38.424714 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:41.138699 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.138669 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:02:41.161341 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.160292 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:02:41.193597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.193572 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:02:41.202201 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.202179 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:02:41.271731 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.271922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271763 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.271922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271827 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.271922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271847 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.271922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271874 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.271922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271891 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.271922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271912 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.271979 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272003 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272036 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272056 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272103 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy\") pod \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\" (UID: \"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272142 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272183 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272239 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272267 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272333 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272368 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert\") pod \"504b8cce-d968-45fc-9b15-508515874fed\" (UID: \"504b8cce-d968-45fc-9b15-508515874fed\") " Nov 01 13:02:41.272762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272737 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.272963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.272916 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.273161 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.273172 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.273400 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.273846 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.274147 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.274469 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.274723 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.275733 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.275747 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.275866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.275809 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.276100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.275977 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:02:41.278524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.278272 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "kube-api-access-r2k7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.278524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.278369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:02:41.278524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.278427 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.278524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.278499 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:02:41.278812 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.278793 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token" pod "504b8cce-d968-45fc-9b15-508515874fed" (UID: "504b8cce-d968-45fc-9b15-508515874fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.280021 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.280001 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld" pod "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" (UID: "8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5"). InnerVolumeSpecName "kube-api-access-zwbld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.373973 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.373893 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.373973 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.373935 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.373973 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.373971 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374005 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374046 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374093 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374119 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374150 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374178 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.374257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374181 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374278 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374429 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374437 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374445 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374508 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374510 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374458 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374574 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.374665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374636 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374672 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374702 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374744 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374772 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374797 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374844 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy\") pod \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\" (UID: \"1b411b1e-c2ba-46fb-8355-d265c9ac6509\") " Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374856 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:02:41.375161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.374904 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data\") pod \"387f169d-f247-488d-851d-509b9a889943\" (UID: \"387f169d-f247-488d-851d-509b9a889943\") " Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375444 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375469 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375483 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375496 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375509 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375525 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375540 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375555 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375578 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504b8cce-d968-45fc-9b15-508515874fed-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375592 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2k7w\" (UniqueName: \"kubernetes.io/projected/504b8cce-d968-45fc-9b15-508515874fed-kube-api-access-r2k7w\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375593 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375609 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387f169d-f247-488d-851d-509b9a889943-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375624 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504b8cce-d968-45fc-9b15-508515874fed-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375638 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375656 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375672 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.375678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375686 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375701 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375715 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375730 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375745 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375760 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbld\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-kube-api-access-zwbld\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375776 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504b8cce-d968-45fc-9b15-508515874fed-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375791 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375807 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375823 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.376583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.375836 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.377114 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377068 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "kube-api-access-v9bfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.377287 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377264 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.377395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377332 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:02:41.377448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377433 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.377649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377623 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:02:41.377754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377721 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "kube-api-access-bpllx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.377921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.377902 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token" pod "387f169d-f247-488d-851d-509b9a889943" (UID: "387f169d-f247-488d-851d-509b9a889943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:02:41.378523 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.378507 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy" pod "1b411b1e-c2ba-46fb-8355-d265c9ac6509" (UID: "1b411b1e-c2ba-46fb-8355-d265c9ac6509"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:02:41.477376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477336 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477365 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477379 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477392 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpllx\" (UniqueName: \"kubernetes.io/projected/387f169d-f247-488d-851d-509b9a889943-kube-api-access-bpllx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477406 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477417 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387f169d-f247-488d-851d-509b9a889943-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477430 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411b1e-c2ba-46fb-8355-d265c9ac6509-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477444 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bfb\" (UniqueName: \"kubernetes.io/projected/1b411b1e-c2ba-46fb-8355-d265c9ac6509-kube-api-access-v9bfb\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.477629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.477458 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387f169d-f247-488d-851d-509b9a889943-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:02:41.580737 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.580697 4484 generic.go:356] "Generic (PLEG): container finished" podID="504b8cce-d968-45fc-9b15-508515874fed" containerID="6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9" exitCode=0 Nov 01 13:02:41.580935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.580761 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" Nov 01 13:02:41.580935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.580777 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" event={"ID":"504b8cce-d968-45fc-9b15-508515874fed","Type":"ContainerDied","Data":"6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9"} Nov 01 13:02:41.580935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.580832 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw" event={"ID":"504b8cce-d968-45fc-9b15-508515874fed","Type":"ContainerDied","Data":"5346458be1d9a53aef59d4a075123763a1d01115cb21edb469bf9b02f3f33be3"} Nov 01 13:02:41.580935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.580858 4484 scope.go:122] "RemoveContainer" containerID="6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9" Nov 01 13:02:41.582353 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.582283 4484 generic.go:356] "Generic (PLEG): container finished" podID="387f169d-f247-488d-851d-509b9a889943" containerID="e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa" exitCode=0 Nov 01 13:02:41.582353 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.582338 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" event={"ID":"387f169d-f247-488d-851d-509b9a889943","Type":"ContainerDied","Data":"e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa"} Nov 01 13:02:41.582489 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.582370 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" Nov 01 13:02:41.582489 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.582392 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76" event={"ID":"387f169d-f247-488d-851d-509b9a889943","Type":"ContainerDied","Data":"b2c2ceca404779736c5545a64a182cd9fc6121ca800d96fd9ce5307d55af904d"} Nov 01 13:02:41.583983 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.583889 4484 generic.go:356] "Generic (PLEG): container finished" podID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerID="35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb" exitCode=0 Nov 01 13:02:41.583983 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.583954 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" Nov 01 13:02:41.583983 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.583973 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" event={"ID":"1b411b1e-c2ba-46fb-8355-d265c9ac6509","Type":"ContainerDied","Data":"35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb"} Nov 01 13:02:41.584206 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.584019 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94" event={"ID":"1b411b1e-c2ba-46fb-8355-d265c9ac6509","Type":"ContainerDied","Data":"e73de69a580809b3f6fd68ba53b7b6cbcb6a330f53f1a03db4a32cf049cad063"} Nov 01 13:02:41.585690 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.585664 4484 generic.go:356] "Generic (PLEG): container finished" podID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerID="15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b" exitCode=0 Nov 01 13:02:41.585792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.585773 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" event={"ID":"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5","Type":"ContainerDied","Data":"15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b"} Nov 01 13:02:41.585860 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.585811 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" event={"ID":"8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5","Type":"ContainerDied","Data":"899046614c14be313e614650116cf27045520ed89d81cacd429fdf7b3bb1e974"} Nov 01 13:02:41.585917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.585879 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh" Nov 01 13:02:41.591622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.591604 4484 scope.go:122] "RemoveContainer" containerID="6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9" Nov 01 13:02:41.591973 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:41.591948 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9\": container with ID starting with 6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9 not found: ID does not exist" containerID="6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9" Nov 01 13:02:41.592065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.591984 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9"} err="failed to get container status \"6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9\": rpc error: code = NotFound desc = could not find container \"6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9\": container with ID starting with 6f601d5be4debb938945f0382c26b12f168a7980f04973c12773b2ea27d383c9 not found: ID does not exist" Nov 01 13:02:41.592065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.592011 4484 scope.go:122] "RemoveContainer" containerID="e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa" Nov 01 13:02:41.602442 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.602425 4484 scope.go:122] "RemoveContainer" containerID="e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa" Nov 01 13:02:41.602653 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:41.602635 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa\": container with ID starting with e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa not found: ID does not exist" containerID="e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa" Nov 01 13:02:41.602712 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.602659 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa"} err="failed to get container status \"e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa\": rpc error: code = NotFound desc = could not find container \"e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa\": container with ID starting with e15a6125abec0cbf972fec2324c31215de39853c6c58efacebb2be247edfb5aa not found: ID does not exist" Nov 01 13:02:41.602712 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.602675 4484 scope.go:122] "RemoveContainer" containerID="35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb" Nov 01 13:02:41.611294 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.611272 4484 scope.go:122] "RemoveContainer" containerID="35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb" Nov 01 13:02:41.611582 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:41.611556 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb\": container with ID starting with 35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb not found: ID does not exist" containerID="35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb" Nov 01 13:02:41.611665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.611591 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb"} err="failed to get container status \"35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb\": rpc error: code = NotFound desc = could not find container \"35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb\": container with ID starting with 35ba4c2349f17345371a6c2f6983bc9c6f965850426959aaef24cfd0ce4985fb not found: ID does not exist" Nov 01 13:02:41.611665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.611612 4484 scope.go:122] "RemoveContainer" containerID="15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b" Nov 01 13:02:41.620947 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.620913 4484 scope.go:122] "RemoveContainer" containerID="15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b" Nov 01 13:02:41.621496 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:02:41.621472 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b\": container with ID starting with 15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b not found: ID does not exist" containerID="15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b" Nov 01 13:02:41.621588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.621504 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b"} err="failed to get container status \"15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b\": rpc error: code = NotFound desc = could not find container \"15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b\": container with ID starting with 15044256bf0dc3baae767e0d955dfaf0bda61c82116d78e40b9df2f8a920ec6b not found: ID does not exist" Nov 01 13:02:41.722742 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.722645 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94"] Nov 01 13:02:41.725353 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.725327 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-m6g94"] Nov 01 13:02:41.828533 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.828505 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh"] Nov 01 13:02:41.836844 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.836818 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wwhnh"] Nov 01 13:02:41.939410 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.939378 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw"] Nov 01 13:02:41.942744 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:41.942722 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-vfbjw"] Nov 01 13:02:42.047213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.047182 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76"] Nov 01 13:02:42.052213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.052178 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-tql76"] Nov 01 13:02:42.368126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.368050 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" path="/var/lib/kubelet/pods/1b411b1e-c2ba-46fb-8355-d265c9ac6509/volumes" Nov 01 13:02:42.368551 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.368535 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387f169d-f247-488d-851d-509b9a889943" path="/var/lib/kubelet/pods/387f169d-f247-488d-851d-509b9a889943/volumes" Nov 01 13:02:42.368933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.368921 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504b8cce-d968-45fc-9b15-508515874fed" path="/var/lib/kubelet/pods/504b8cce-d968-45fc-9b15-508515874fed/volumes" Nov 01 13:02:42.369627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.369612 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" path="/var/lib/kubelet/pods/8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5/volumes" Nov 01 13:02:42.928556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:42.928526 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:43.074077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:43.074047 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:43.791639 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:43.791611 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:49.089511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:49.089477 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:02:49.118399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:49.118370 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:52.058835 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:52.055960 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2"] Nov 01 13:02:52.068327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:52.066928 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b56a7d-nxdj2"] Nov 01 13:02:52.367924 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:52.367849 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4967d4d-cda5-4e04-a2e6-a69b6da666fc" path="/var/lib/kubelet/pods/c4967d4d-cda5-4e04-a2e6-a69b6da666fc/volumes" Nov 01 13:02:54.509411 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.509380 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:02:54.593333 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.593269 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz"] Nov 01 13:02:54.593978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.593951 4484 cpu_manager.go:414] "containerMap: removing container" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" Nov 01 13:02:54.593978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.593979 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.593998 4484 cpu_manager.go:414] "containerMap: removing container" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594006 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594025 4484 cpu_manager.go:414] "containerMap: removing container" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594033 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594043 4484 cpu_manager.go:414] "containerMap: removing container" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594050 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594069 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594075 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594086 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594093 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594103 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594110 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594124 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3c9ada65-8080-43ed-aadb-d5fc97e024b5" containerName="revoke-keys" Nov 01 13:02:54.594190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594132 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9ada65-8080-43ed-aadb-d5fc97e024b5" containerName="revoke-keys" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594231 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="504b8cce-d968-45fc-9b15-508515874fed" containerName="istio-proxy" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594242 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ed692d-230c-4d02-8475-eec80bab204f" containerName="main" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594254 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0cd339-c08e-40c9-a5a1-7ba0e791b0c5" containerName="istio-proxy" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594263 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="387f169d-f247-488d-851d-509b9a889943" containerName="istio-proxy" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594273 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594282 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b411b1e-c2ba-46fb-8355-d265c9ac6509" containerName="istio-proxy" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594289 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a4d823-1015-41d5-87d5-962ea45b4ae7" containerName="istio-proxy" Nov 01 13:02:54.595065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.594312 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9ada65-8080-43ed-aadb-d5fc97e024b5" containerName="revoke-keys" Nov 01 13:02:54.600438 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.600398 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.682919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.682888 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz"] Nov 01 13:02:54.707422 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707390 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwwj\" (UniqueName: \"kubernetes.io/projected/75f922a6-c170-4d88-a0e5-45c2b4e6b763-kube-api-access-9dwwj\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707468 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707517 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707558 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707784 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707744 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707820 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707860 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.707905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.707886 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.708073 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.708047 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.808775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.808700 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.808775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.808741 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.808775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.808767 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809032 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.808804 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809032 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.808825 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809032 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.808972 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809139 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809060 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwwj\" (UniqueName: \"kubernetes.io/projected/75f922a6-c170-4d88-a0e5-45c2b4e6b763-kube-api-access-9dwwj\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809139 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809117 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809165 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809555 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809697 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809633 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809709 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809890 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809871 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.809987 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.809967 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.811564 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.811537 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.812451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.812426 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.863738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.863718 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f922a6-c170-4d88-a0e5-45c2b4e6b763-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.863829 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.863783 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwwj\" (UniqueName: \"kubernetes.io/projected/75f922a6-c170-4d88-a0e5-45c2b4e6b763-kube-api-access-9dwwj\") pod \"maas-default-gateway-openshift-default-7df854bf67-gkqlz\" (UID: \"75f922a6-c170-4d88-a0e5-45c2b4e6b763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:54.913593 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:54.913559 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:55.348115 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.348090 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz"] Nov 01 13:02:55.350466 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:02:55.350421 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f922a6_c170_4d88_a0e5_45c2b4e6b763.slice/crio-31afe627ca47c737b17e0c7cfcf14ab52ce75843de04d487ea8c313da93341b8 WatchSource:0}: Error finding container 31afe627ca47c737b17e0c7cfcf14ab52ce75843de04d487ea8c313da93341b8: Status 404 returned error can't find the container with id 31afe627ca47c737b17e0c7cfcf14ab52ce75843de04d487ea8c313da93341b8 Nov 01 13:02:55.352801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.352772 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:02:55.352882 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.352854 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:02:55.352927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.352881 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:02:55.680454 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.680326 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" event={"ID":"75f922a6-c170-4d88-a0e5-45c2b4e6b763","Type":"ContainerStarted","Data":"ca15572ba8cd1dcab2a7ade5aef8ebbc64c2493c92c72b5a6994a82da38edb23"} Nov 01 13:02:55.680454 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.680405 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" event={"ID":"75f922a6-c170-4d88-a0e5-45c2b4e6b763","Type":"ContainerStarted","Data":"31afe627ca47c737b17e0c7cfcf14ab52ce75843de04d487ea8c313da93341b8"} Nov 01 13:02:55.793866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.793808 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podStartSLOduration=1.79379327 podStartE2EDuration="1.79379327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:02:55.791823388 +0000 UTC m=+2445.541766365" watchObservedRunningTime="2026-11-01 13:02:55.79379327 +0000 UTC m=+2445.543736320" Nov 01 13:02:55.914611 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:55.914584 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:02:56.915915 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:56.915881 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 13:02:56.916355 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:56.915929 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 13:02:57.914753 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:57.914721 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 13:02:57.914931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:57.914772 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 13:02:58.914297 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:58.914265 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 13:02:58.914710 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:58.914343 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 13:02:59.914027 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:59.913984 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 13:02:59.914246 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:02:59.914057 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 13:03:00.031462 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:00.031402 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:00.274244 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:03:00.274214 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:03:00.913841 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:00.913804 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 13:03:00.914041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:00.913856 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 13:03:01.913854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:01.913821 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 13:03:01.914357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:01.913875 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 13:03:03.729421 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:03.729385 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:03:03.730034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:03.730011 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:03:03.730868 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:03.730847 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" Nov 01 13:03:05.393076 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:05.393026 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:10.697784 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:10.697734 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:11.190518 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:11.190472 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:11.766221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:11.766192 4484 scope.go:122] "RemoveContainer" containerID="bb41658818197d2b3520cfc85672a72e1c6e67cd1859dbe2aa87f4ea599062a0" Nov 01 13:03:11.775226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:11.775208 4484 scope.go:122] "RemoveContainer" containerID="995a5674297bbc04c3d976b04c477962ea03cf1d01573c14bba3385c9f51d4a6" Nov 01 13:03:11.783467 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:11.783445 4484 scope.go:122] "RemoveContainer" containerID="31ad5f05a836573438becd641f3781d0cdefcf9c6aa46e3dd941a07eadeff835" Nov 01 13:03:13.865029 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:13.865002 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:16.030153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:16.030119 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:21.375824 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:21.375795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:24.313653 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:24.313580 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:24.931812 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:24.931779 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:03:26.939417 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:26.939386 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:28.326224 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:28.326188 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:28.439472 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:28.439439 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:29.294668 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:29.294632 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99"] Nov 01 13:03:29.341186 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:29.341151 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn"] Nov 01 13:03:29.343651 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:29.343620 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf"] Nov 01 13:03:29.383431 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:29.383404 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:30.250654 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:03:30.250619 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:03:32.501607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:32.501568 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:35.516594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:35.516561 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:36.438695 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:36.438654 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75"] Nov 01 13:03:36.552915 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:36.552885 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892"] Nov 01 13:03:36.798334 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:36.791190 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:36.945226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:36.945191 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:37.932639 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:37.932588 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:41.439869 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:41.439818 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerName="payload-processing" containerID="cri-o://0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409" gracePeriod=25 Nov 01 13:03:41.554572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:41.554523 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerName="payload-pre-processing" containerID="cri-o://f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe" gracePeriod=25 Nov 01 13:03:43.251560 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:43.251525 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:43.326886 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:43.326831 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:03:44.308949 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.308915 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:03:44.314380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.314350 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" containerID="cri-o://136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18" gracePeriod=45 Nov 01 13:03:44.360443 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.360412 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" containerID="cri-o://40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222" gracePeriod=45 Nov 01 13:03:44.362364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.362318 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" containerID="cri-o://6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511" gracePeriod=45 Nov 01 13:03:44.382405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.382372 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:44.639285 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.639263 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 13:03:44.641333 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.641285 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 13:03:44.656249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.656229 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 13:03:44.801057 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801030 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm\") pod \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " Nov 01 13:03:44.801249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801083 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir\") pod \"3419f076-07d5-4793-812b-b95a40449811\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " Nov 01 13:03:44.801249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801124 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs\") pod \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " Nov 01 13:03:44.801249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801155 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home\") pod \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " Nov 01 13:03:44.801249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801182 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home\") pod \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " Nov 01 13:03:44.801249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801209 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg\" (UniqueName: \"kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg\") pod \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801264 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm\") pod \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801333 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm\") pod \"3419f076-07d5-4793-812b-b95a40449811\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801366 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache\") pod \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801413 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8\" (UniqueName: \"kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8\") pod \"3419f076-07d5-4793-812b-b95a40449811\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801434 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir" pod "3419f076-07d5-4793-812b-b95a40449811" (UID: "3419f076-07d5-4793-812b-b95a40449811"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801449 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home" pod "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" (UID: "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801445 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir\") pod \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " Nov 01 13:03:44.801559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801532 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home\") pod \"3419f076-07d5-4793-812b-b95a40449811\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801596 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache\") pod \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801628 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir" pod "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" (UID: "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801652 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs\" (UniqueName: \"kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs\") pod \"3419f076-07d5-4793-812b-b95a40449811\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801699 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj\" (UniqueName: \"kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj\") pod \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801746 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache\") pod \"3419f076-07d5-4793-812b-b95a40449811\" (UID: \"3419f076-07d5-4793-812b-b95a40449811\") " Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801780 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs\") pod \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\" (UID: \"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3\") " Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801809 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home" pod "7adbc4ea-a43d-4441-8cda-9fed508bfbda" (UID: "7adbc4ea-a43d-4441-8cda-9fed508bfbda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801825 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache" pod "7adbc4ea-a43d-4441-8cda-9fed508bfbda" (UID: "7adbc4ea-a43d-4441-8cda-9fed508bfbda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.801979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.801820 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir\") pod \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\" (UID: \"7adbc4ea-a43d-4441-8cda-9fed508bfbda\") " Nov 01 13:03:44.802501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802009 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir" pod "7adbc4ea-a43d-4441-8cda-9fed508bfbda" (UID: "7adbc4ea-a43d-4441-8cda-9fed508bfbda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.802501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802199 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home" pod "3419f076-07d5-4793-812b-b95a40449811" (UID: "3419f076-07d5-4793-812b-b95a40449811"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.802501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802385 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache" pod "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" (UID: "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.802501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802454 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802477 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802501 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802492 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802505 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802518 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802531 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802544 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.802830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.802753 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache" pod "3419f076-07d5-4793-812b-b95a40449811" (UID: "3419f076-07d5-4793-812b-b95a40449811"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.804464 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.804437 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm" pod "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" (UID: "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.804770 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.804745 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm" pod "3419f076-07d5-4793-812b-b95a40449811" (UID: "3419f076-07d5-4793-812b-b95a40449811"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.804930 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.804829 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm" pod "7adbc4ea-a43d-4441-8cda-9fed508bfbda" (UID: "7adbc4ea-a43d-4441-8cda-9fed508bfbda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:03:44.805155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.805120 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8" pod "3419f076-07d5-4793-812b-b95a40449811" (UID: "3419f076-07d5-4793-812b-b95a40449811"). InnerVolumeSpecName "kube-api-access-lmbf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:03:44.805266 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.805165 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs" pod "3419f076-07d5-4793-812b-b95a40449811" (UID: "3419f076-07d5-4793-812b-b95a40449811"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:03:44.805366 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.805331 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg" pod "7adbc4ea-a43d-4441-8cda-9fed508bfbda" (UID: "7adbc4ea-a43d-4441-8cda-9fed508bfbda"). InnerVolumeSpecName "kube-api-access-5zdqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:03:44.805473 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.805449 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs" pod "7adbc4ea-a43d-4441-8cda-9fed508bfbda" (UID: "7adbc4ea-a43d-4441-8cda-9fed508bfbda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:03:44.805546 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.805530 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs" pod "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" (UID: "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:03:44.805858 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.805844 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj" pod "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" (UID: "e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3"). InnerVolumeSpecName "kube-api-access-2bnsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:03:44.903819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903745 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbf8\" (UniqueName: \"kubernetes.io/projected/3419f076-07d5-4793-812b-b95a40449811-kube-api-access-lmbf8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.903819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903779 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.903819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903790 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3419f076-07d5-4793-812b-b95a40449811-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.903819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903800 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnsj\" (UniqueName: \"kubernetes.io/projected/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-kube-api-access-2bnsj\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.903819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903809 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.903819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903817 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.904121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903825 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.904121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903834 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4ea-a43d-4441-8cda-9fed508bfbda-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.904121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903842 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdqg\" (UniqueName: \"kubernetes.io/projected/7adbc4ea-a43d-4441-8cda-9fed508bfbda-kube-api-access-5zdqg\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.904121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903850 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7adbc4ea-a43d-4441-8cda-9fed508bfbda-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:44.904121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:44.903858 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3419f076-07d5-4793-812b-b95a40449811-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:03:45.016743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.016700 4484 generic.go:356] "Generic (PLEG): container finished" podID="3419f076-07d5-4793-812b-b95a40449811" containerID="40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222" exitCode=0 Nov 01 13:03:45.016914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.016807 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" Nov 01 13:03:45.016914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.016828 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" event={"ID":"3419f076-07d5-4793-812b-b95a40449811","Type":"ContainerDied","Data":"40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222"} Nov 01 13:03:45.016914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.016867 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn" event={"ID":"3419f076-07d5-4793-812b-b95a40449811","Type":"ContainerDied","Data":"bfaa0732a6cb9d12b54aa13692371794474ab052384643bd51baefd3836af829"} Nov 01 13:03:45.016914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.016886 4484 scope.go:122] "RemoveContainer" containerID="40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222" Nov 01 13:03:45.018429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.018403 4484 generic.go:356] "Generic (PLEG): container finished" podID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerID="136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18" exitCode=0 Nov 01 13:03:45.018555 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.018489 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" Nov 01 13:03:45.018555 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.018501 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" event={"ID":"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3","Type":"ContainerDied","Data":"136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18"} Nov 01 13:03:45.018555 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.018531 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99" event={"ID":"e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3","Type":"ContainerDied","Data":"e4b1a33b14fb8ac84b1a7ff812c2d6920e41a7173835a730378b7146c816372e"} Nov 01 13:03:45.019932 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.019907 4484 generic.go:356] "Generic (PLEG): container finished" podID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerID="6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511" exitCode=0 Nov 01 13:03:45.020028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.019954 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" event={"ID":"7adbc4ea-a43d-4441-8cda-9fed508bfbda","Type":"ContainerDied","Data":"6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511"} Nov 01 13:03:45.020028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.019982 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" event={"ID":"7adbc4ea-a43d-4441-8cda-9fed508bfbda","Type":"ContainerDied","Data":"8aeb8c6bb1a110cd282e72b067461f2e3a64cfaf27376d2e5df85f3cc1eeff0a"} Nov 01 13:03:45.020115 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.020051 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf" Nov 01 13:03:45.029725 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.029707 4484 scope.go:122] "RemoveContainer" containerID="40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222" Nov 01 13:03:45.030001 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:03:45.029983 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222\": container with ID starting with 40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222 not found: ID does not exist" containerID="40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222" Nov 01 13:03:45.030075 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.030009 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222"} err="failed to get container status \"40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222\": rpc error: code = NotFound desc = could not find container \"40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222\": container with ID starting with 40f50e4543a14d60420b425bd6d7c85221ad902f6319ab3d2b76b580ddacf222 not found: ID does not exist" Nov 01 13:03:45.030075 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.030026 4484 scope.go:122] "RemoveContainer" containerID="136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18" Nov 01 13:03:45.041431 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.041404 4484 scope.go:122] "RemoveContainer" containerID="136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18" Nov 01 13:03:45.041803 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:03:45.041779 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18\": container with ID starting with 136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18 not found: ID does not exist" containerID="136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18" Nov 01 13:03:45.041881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.041814 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18"} err="failed to get container status \"136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18\": rpc error: code = NotFound desc = could not find container \"136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18\": container with ID starting with 136dd66f7bd8728b872eeba46ea57ae539f5124ec5bdeacc6a93072b42a3bb18 not found: ID does not exist" Nov 01 13:03:45.041881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.041839 4484 scope.go:122] "RemoveContainer" containerID="6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511" Nov 01 13:03:45.053419 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.053395 4484 scope.go:122] "RemoveContainer" containerID="6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511" Nov 01 13:03:45.053754 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:03:45.053725 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511\": container with ID starting with 6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511 not found: ID does not exist" containerID="6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511" Nov 01 13:03:45.053833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.053765 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511"} err="failed to get container status \"6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511\": rpc error: code = NotFound desc = could not find container \"6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511\": container with ID starting with 6e5e3702f546a0d57016bedc837dcace343baecee7138ba9521a56c25814e511 not found: ID does not exist" Nov 01 13:03:45.171560 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.171471 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn"] Nov 01 13:03:45.185763 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.185735 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/premium-w3-ecd640-kserve-7557b7b7d9-sptrn"] Nov 01 13:03:45.286931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.286901 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf"] Nov 01 13:03:45.291259 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.291232 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/simulator-w3-ecd640-kserve-8478c6d8cb-b8czf"] Nov 01 13:03:45.393213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.393185 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99"] Nov 01 13:03:45.398682 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:45.398660 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-w3-ecd640-kserve-5d999d4c76-wqc99"] Nov 01 13:03:46.235313 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:46.235268 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:03:46.368169 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:46.368135 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3419f076-07d5-4793-812b-b95a40449811" path="/var/lib/kubelet/pods/3419f076-07d5-4793-812b-b95a40449811/volumes" Nov 01 13:03:46.368587 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:46.368572 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" path="/var/lib/kubelet/pods/7adbc4ea-a43d-4441-8cda-9fed508bfbda/volumes" Nov 01 13:03:46.368954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:46.368942 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" path="/var/lib/kubelet/pods/e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3/volumes" Nov 01 13:03:49.095165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:49.095132 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:49.547539 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:49.547508 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:53.304131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:53.304096 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:03:54.309060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:54.309018 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:03:54.829860 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:54.829831 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:56.479596 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:56.479543 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs"] Nov 01 13:03:56.828960 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:56.828864 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:03:56.947540 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:03:56.947504 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:00.195894 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:00.195862 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:00.268056 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:04:00.267995 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:04:01.480613 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.480564 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" podUID="99915904-ffac-435d-a0a5-da93549daca6" containerName="payload-processing" containerID="cri-o://9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c" gracePeriod=25 Nov 01 13:04:01.692165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.692140 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 13:04:01.702257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.702192 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn\" (UniqueName: \"kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn\") pod \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " Nov 01 13:04:01.702381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.702331 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume\") pod \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\" (UID: \"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339\") " Nov 01 13:04:01.702776 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.702746 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume" pod "7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" (UID: "7a9900f9-e18d-4dc7-a3f1-d7faa0acb339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:01.704111 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.704088 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn" pod "7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" (UID: "7a9900f9-e18d-4dc7-a3f1-d7faa0acb339"). InnerVolumeSpecName "kube-api-access-zd8kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:01.783837 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.783811 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 13:04:01.803578 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.803549 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x\" (UniqueName: \"kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x\") pod \"131ac484-20b8-40a4-bc0d-68cc33f01928\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " Nov 01 13:04:01.803722 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.803642 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume\") pod \"131ac484-20b8-40a4-bc0d-68cc33f01928\" (UID: \"131ac484-20b8-40a4-bc0d-68cc33f01928\") " Nov 01 13:04:01.804057 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.804032 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:01.804186 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.804060 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd8kn\" (UniqueName: \"kubernetes.io/projected/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339-kube-api-access-zd8kn\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:01.804186 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.804065 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume" pod "131ac484-20b8-40a4-bc0d-68cc33f01928" (UID: "131ac484-20b8-40a4-bc0d-68cc33f01928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:01.806253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.806232 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x" pod "131ac484-20b8-40a4-bc0d-68cc33f01928" (UID: "131ac484-20b8-40a4-bc0d-68cc33f01928"). InnerVolumeSpecName "kube-api-access-n262x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:01.905493 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.905463 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n262x\" (UniqueName: \"kubernetes.io/projected/131ac484-20b8-40a4-bc0d-68cc33f01928-kube-api-access-n262x\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:01.905493 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:01.905488 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131ac484-20b8-40a4-bc0d-68cc33f01928-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:02.115554 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.115515 4484 generic.go:356] "Generic (PLEG): container finished" podID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerID="f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe" exitCode=0 Nov 01 13:04:02.115716 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.115571 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" Nov 01 13:04:02.115716 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.115600 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" event={"ID":"131ac484-20b8-40a4-bc0d-68cc33f01928","Type":"ContainerDied","Data":"f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe"} Nov 01 13:04:02.115716 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.115655 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892" event={"ID":"131ac484-20b8-40a4-bc0d-68cc33f01928","Type":"ContainerDied","Data":"5b0de6db88c2b802a9173ecb844e95cd5ddf596f935a65a181be916a2f3f2409"} Nov 01 13:04:02.115716 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.115677 4484 scope.go:122] "RemoveContainer" containerID="f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe" Nov 01 13:04:02.117050 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.117027 4484 generic.go:356] "Generic (PLEG): container finished" podID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerID="0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409" exitCode=0 Nov 01 13:04:02.117165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.117083 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" Nov 01 13:04:02.117165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.117084 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" event={"ID":"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339","Type":"ContainerDied","Data":"0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409"} Nov 01 13:04:02.117244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.117205 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75" event={"ID":"7a9900f9-e18d-4dc7-a3f1-d7faa0acb339","Type":"ContainerDied","Data":"412d3ba3f9bfcf2c8008c2136ecd211b13a9aea3fe6a6098cc46351a1ba1a16f"} Nov 01 13:04:02.128005 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.127987 4484 scope.go:122] "RemoveContainer" containerID="f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe" Nov 01 13:04:02.128269 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:04:02.128250 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe\": container with ID starting with f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe not found: ID does not exist" containerID="f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe" Nov 01 13:04:02.128376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.128280 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe"} err="failed to get container status \"f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe\": rpc error: code = NotFound desc = could not find container \"f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe\": container with ID starting with f0912393bd094187c41a5050b09239cb0db0b4edf09c90354dbc6c9b593d03fe not found: ID does not exist" Nov 01 13:04:02.128376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.128317 4484 scope.go:122] "RemoveContainer" containerID="0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409" Nov 01 13:04:02.136281 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.136260 4484 scope.go:122] "RemoveContainer" containerID="0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409" Nov 01 13:04:02.136571 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:04:02.136548 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409\": container with ID starting with 0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409 not found: ID does not exist" containerID="0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409" Nov 01 13:04:02.136641 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.136577 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409"} err="failed to get container status \"0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409\": rpc error: code = NotFound desc = could not find container \"0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409\": container with ID starting with 0412872ca170a562012f91d2b8966de0b3ba3fc73144c652773b06b4ca0b5409 not found: ID does not exist" Nov 01 13:04:02.268179 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.268157 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892"] Nov 01 13:04:02.314570 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.314537 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ecd640-6c488c647-qt892"] Nov 01 13:04:02.368597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.368530 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" path="/var/lib/kubelet/pods/131ac484-20b8-40a4-bc0d-68cc33f01928/volumes" Nov 01 13:04:02.415970 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.415947 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75"] Nov 01 13:04:02.420962 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:02.420937 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ecd640-7f54c45b77-j7f75"] Nov 01 13:04:04.367628 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:04.367593 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" path="/var/lib/kubelet/pods/7a9900f9-e18d-4dc7-a3f1-d7faa0acb339/volumes" Nov 01 13:04:05.014132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014098 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s"] Nov 01 13:04:05.014580 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014562 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" Nov 01 13:04:05.014580 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014581 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014598 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014604 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014615 4484 cpu_manager.go:414] "containerMap: removing container" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerName="payload-processing" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014621 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerName="payload-processing" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014639 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014644 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014669 4484 cpu_manager.go:414] "containerMap: removing container" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerName="payload-pre-processing" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014674 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerName="payload-pre-processing" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014738 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3419f076-07d5-4793-812b-b95a40449811" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014751 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="131ac484-20b8-40a4-bc0d-68cc33f01928" containerName="payload-pre-processing" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014760 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adbc4ea-a43d-4441-8cda-9fed508bfbda" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014767 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14ba1ec-ab50-4e1c-8418-725fb7b1d7a3" containerName="main" Nov 01 13:04:05.014764 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.014773 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9900f9-e18d-4dc7-a3f1-d7faa0acb339" containerName="payload-processing" Nov 01 13:04:05.019509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.019487 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.035286 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.035264 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dwq\" (UniqueName: \"kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.035404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.035326 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.035525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.035501 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.071359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.071331 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s"] Nov 01 13:04:05.114479 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.114454 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:04:05.114606 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.114495 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:04:05.136940 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.136915 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dwq\" (UniqueName: \"kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.137059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.136953 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.137059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.137020 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.137665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.137648 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.139280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.139261 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.190631 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.190605 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dwq\" (UniqueName: \"kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq\") pod \"maas-api-revoke-sub-dbdc138a-cdl5s\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.336126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.336061 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:05.715041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.715015 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s"] Nov 01 13:04:05.717939 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:04:05.717904 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9db29de_e28a_42b1_a896_9c82cd36a607.slice/crio-f1e0d90c3f1f8beb4fe6b3f105eaceac6483f7e3537a114fad1d40610de2b451 WatchSource:0}: Error finding container f1e0d90c3f1f8beb4fe6b3f105eaceac6483f7e3537a114fad1d40610de2b451: Status 404 returned error can't find the container with id f1e0d90c3f1f8beb4fe6b3f105eaceac6483f7e3537a114fad1d40610de2b451 Nov 01 13:04:05.887843 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:05.887811 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:06.145036 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:06.145002 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" event={"ID":"b9db29de-e28a-42b1-a896-9c82cd36a607","Type":"ContainerStarted","Data":"901482421abbfac7f4619cf8307c6b7d0f550dd82d40d5f06da320db00c54b4f"} Nov 01 13:04:06.145216 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:06.145054 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" event={"ID":"b9db29de-e28a-42b1-a896-9c82cd36a607","Type":"ContainerStarted","Data":"f1e0d90c3f1f8beb4fe6b3f105eaceac6483f7e3537a114fad1d40610de2b451"} Nov 01 13:04:06.306613 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:06.306553 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" podStartSLOduration=2.306539087 podStartE2EDuration="2.306539087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:04:06.303221234 +0000 UTC m=+2516.053164213" watchObservedRunningTime="2026-11-01 13:04:06.306539087 +0000 UTC m=+2516.056482065" Nov 01 13:04:06.385064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:06.385024 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" podUID="99915904-ffac-435d-a0a5-da93549daca6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:04:06.477768 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:06.477684 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:07.009552 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:07.009519 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:07.130773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:07.130740 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:07.152314 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:07.152268 4484 generic.go:356] "Generic (PLEG): container finished" podID="b9db29de-e28a-42b1-a896-9c82cd36a607" containerID="901482421abbfac7f4619cf8307c6b7d0f550dd82d40d5f06da320db00c54b4f" exitCode=0 Nov 01 13:04:07.152448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:07.152330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" event={"ID":"b9db29de-e28a-42b1-a896-9c82cd36a607","Type":"ContainerDied","Data":"901482421abbfac7f4619cf8307c6b7d0f550dd82d40d5f06da320db00c54b4f"} Nov 01 13:04:07.461510 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:07.461431 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:08.296994 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.296975 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:08.369019 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.368995 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca\") pod \"b9db29de-e28a-42b1-a896-9c82cd36a607\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " Nov 01 13:04:08.369121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.369081 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq\" (UniqueName: \"kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq\") pod \"b9db29de-e28a-42b1-a896-9c82cd36a607\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " Nov 01 13:04:08.369121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.369110 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca\") pod \"b9db29de-e28a-42b1-a896-9c82cd36a607\" (UID: \"b9db29de-e28a-42b1-a896-9c82cd36a607\") " Nov 01 13:04:08.369292 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.369274 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca" pod "b9db29de-e28a-42b1-a896-9c82cd36a607" (UID: "b9db29de-e28a-42b1-a896-9c82cd36a607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:08.369415 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.369400 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:08.371031 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.371004 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq" pod "b9db29de-e28a-42b1-a896-9c82cd36a607" (UID: "b9db29de-e28a-42b1-a896-9c82cd36a607"). InnerVolumeSpecName "kube-api-access-j2dwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:08.371128 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.371065 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca" pod "b9db29de-e28a-42b1-a896-9c82cd36a607" (UID: "b9db29de-e28a-42b1-a896-9c82cd36a607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:04:08.470712 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.470636 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dwq\" (UniqueName: \"kubernetes.io/projected/b9db29de-e28a-42b1-a896-9c82cd36a607-kube-api-access-j2dwq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:08.470712 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:08.470664 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9db29de-e28a-42b1-a896-9c82cd36a607-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:09.165392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:09.165361 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" Nov 01 13:04:09.165651 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:09.165403 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s" event={"ID":"b9db29de-e28a-42b1-a896-9c82cd36a607","Type":"ContainerDied","Data":"f1e0d90c3f1f8beb4fe6b3f105eaceac6483f7e3537a114fad1d40610de2b451"} Nov 01 13:04:09.165651 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:09.165448 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e0d90c3f1f8beb4fe6b3f105eaceac6483f7e3537a114fad1d40610de2b451" Nov 01 13:04:11.256253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:11.256220 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:11.837642 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:11.837608 4484 scope.go:122] "RemoveContainer" containerID="70cef6b8c957859a7478afdfa7e9ace7e904e93e93aa005e78bcd6cfb6dfe291" Nov 01 13:04:11.846660 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:11.846629 4484 scope.go:122] "RemoveContainer" containerID="f37803a80b7b29810d4b8751c3f4e4d289c4179450e9f1e5903c6c8a7ecc7518" Nov 01 13:04:13.152741 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:13.152712 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:13.275996 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:13.275968 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:14.221475 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:14.221444 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:14.536877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:14.536845 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:04:16.385248 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:16.385210 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" podUID="99915904-ffac-435d-a0a5-da93549daca6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:04:16.587933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:16.587900 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:21.408316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.408268 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt"] Nov 01 13:04:21.408959 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.408937 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b9db29de-e28a-42b1-a896-9c82cd36a607" containerName="revoke-keys" Nov 01 13:04:21.409009 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.408963 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9db29de-e28a-42b1-a896-9c82cd36a607" containerName="revoke-keys" Nov 01 13:04:21.409124 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.409111 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9db29de-e28a-42b1-a896-9c82cd36a607" containerName="revoke-keys" Nov 01 13:04:21.414415 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.414388 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.463946 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.463910 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt"] Nov 01 13:04:21.506905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.506873 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:04:21.507102 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.507080 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:04:21.507230 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.507210 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.507283 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.507269 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.507508 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.507451 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4f6d\" (UniqueName: \"kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.608714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.608684 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.608866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.608739 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.608866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.608829 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4f6d\" (UniqueName: \"kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.609415 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.609389 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.611019 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.610994 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.662490 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.662420 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4f6d\" (UniqueName: \"kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d\") pod \"maas-api-revoke-sub-95943f4b-hgtgt\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.728391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.728366 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 13:04:21.735422 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.735406 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:21.810969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.810926 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp\" (UniqueName: \"kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp\") pod \"99915904-ffac-435d-a0a5-da93549daca6\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " Nov 01 13:04:21.811105 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.810977 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume\") pod \"99915904-ffac-435d-a0a5-da93549daca6\" (UID: \"99915904-ffac-435d-a0a5-da93549daca6\") " Nov 01 13:04:21.811534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.811500 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume" pod "99915904-ffac-435d-a0a5-da93549daca6" (UID: "99915904-ffac-435d-a0a5-da93549daca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:21.813367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.813339 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp" pod "99915904-ffac-435d-a0a5-da93549daca6" (UID: "99915904-ffac-435d-a0a5-da93549daca6"). InnerVolumeSpecName "kube-api-access-9krmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:21.912471 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.912441 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9krmp\" (UniqueName: \"kubernetes.io/projected/99915904-ffac-435d-a0a5-da93549daca6-kube-api-access-9krmp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:21.912471 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:21.912466 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99915904-ffac-435d-a0a5-da93549daca6-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:22.100377 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.100353 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt"] Nov 01 13:04:22.102711 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:04:22.102677 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98def0f2_4fae_4bd7_8a33_f288eef96e2d.slice/crio-dbf928baa4f6c7f254ab47ca1438e7301651e801faab82fcd6e59638f1d7dc2d WatchSource:0}: Error finding container dbf928baa4f6c7f254ab47ca1438e7301651e801faab82fcd6e59638f1d7dc2d: Status 404 returned error can't find the container with id dbf928baa4f6c7f254ab47ca1438e7301651e801faab82fcd6e59638f1d7dc2d Nov 01 13:04:22.233847 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.233812 4484 generic.go:356] "Generic (PLEG): container finished" podID="99915904-ffac-435d-a0a5-da93549daca6" containerID="9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c" exitCode=0 Nov 01 13:04:22.234024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.233863 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" Nov 01 13:04:22.234024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.233892 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" event={"ID":"99915904-ffac-435d-a0a5-da93549daca6","Type":"ContainerDied","Data":"9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c"} Nov 01 13:04:22.234024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.233951 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs" event={"ID":"99915904-ffac-435d-a0a5-da93549daca6","Type":"ContainerDied","Data":"1f48be52baa9de1dae9c4f117db5d870838c5aad1df10d09d41de45a4350ba4f"} Nov 01 13:04:22.234024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.233976 4484 scope.go:122] "RemoveContainer" containerID="9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c" Nov 01 13:04:22.235228 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.235208 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" event={"ID":"98def0f2-4fae-4bd7-8a33-f288eef96e2d","Type":"ContainerStarted","Data":"dbf928baa4f6c7f254ab47ca1438e7301651e801faab82fcd6e59638f1d7dc2d"} Nov 01 13:04:22.244362 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.244345 4484 scope.go:122] "RemoveContainer" containerID="9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c" Nov 01 13:04:22.244635 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:04:22.244614 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c\": container with ID starting with 9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c not found: ID does not exist" containerID="9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c" Nov 01 13:04:22.244693 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.244646 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c"} err="failed to get container status \"9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c\": rpc error: code = NotFound desc = could not find container \"9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c\": container with ID starting with 9bb64a883ad78a347f29cfc57451b5aeae038dd200d8d4a5c5985b9d4852203c not found: ID does not exist" Nov 01 13:04:22.300429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.300404 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:22.389385 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.389356 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs"] Nov 01 13:04:22.390074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:22.390055 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-06fd6b-7d6b54664c-zhmbs"] Nov 01 13:04:23.243061 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:23.243028 4484 generic.go:356] "Generic (PLEG): container finished" podID="98def0f2-4fae-4bd7-8a33-f288eef96e2d" containerID="edeafffd3507a81a0f5c9624d65aae26844d12a68713e1189c5b13efe153c4ad" exitCode=0 Nov 01 13:04:23.243498 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:23.243083 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" event={"ID":"98def0f2-4fae-4bd7-8a33-f288eef96e2d","Type":"ContainerDied","Data":"edeafffd3507a81a0f5c9624d65aae26844d12a68713e1189c5b13efe153c4ad"} Nov 01 13:04:24.369458 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.369426 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99915904-ffac-435d-a0a5-da93549daca6" path="/var/lib/kubelet/pods/99915904-ffac-435d-a0a5-da93549daca6/volumes" Nov 01 13:04:24.377409 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.377387 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:24.537476 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.537398 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca\") pod \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " Nov 01 13:04:24.537746 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.537483 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca\") pod \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " Nov 01 13:04:24.537746 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.537561 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d\" (UniqueName: \"kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d\") pod \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\" (UID: \"98def0f2-4fae-4bd7-8a33-f288eef96e2d\") " Nov 01 13:04:24.537934 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.537905 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca" pod "98def0f2-4fae-4bd7-8a33-f288eef96e2d" (UID: "98def0f2-4fae-4bd7-8a33-f288eef96e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:24.539555 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.539533 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca" pod "98def0f2-4fae-4bd7-8a33-f288eef96e2d" (UID: "98def0f2-4fae-4bd7-8a33-f288eef96e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:04:24.539650 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.539559 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d" pod "98def0f2-4fae-4bd7-8a33-f288eef96e2d" (UID: "98def0f2-4fae-4bd7-8a33-f288eef96e2d"). InnerVolumeSpecName "kube-api-access-n4f6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:24.638591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.638558 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:24.638591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.638586 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4f6d\" (UniqueName: \"kubernetes.io/projected/98def0f2-4fae-4bd7-8a33-f288eef96e2d-kube-api-access-n4f6d\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:24.638591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:24.638597 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98def0f2-4fae-4bd7-8a33-f288eef96e2d-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:25.255147 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:25.255113 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" Nov 01 13:04:25.255340 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:25.255147 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt" event={"ID":"98def0f2-4fae-4bd7-8a33-f288eef96e2d","Type":"ContainerDied","Data":"dbf928baa4f6c7f254ab47ca1438e7301651e801faab82fcd6e59638f1d7dc2d"} Nov 01 13:04:25.255340 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:25.255197 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf928baa4f6c7f254ab47ca1438e7301651e801faab82fcd6e59638f1d7dc2d" Nov 01 13:04:26.966408 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:26.966371 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:27.097577 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:27.097543 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:27.295884 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:27.295853 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:27.782609 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:27.782575 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:30.251526 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:04:30.251488 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:04:32.458032 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:32.457990 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:32.984954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:32.984919 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:34.150018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:34.149969 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:34.411227 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:34.411113 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:34.534971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:34.531778 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:35.302188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:35.302132 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:38.619954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:38.619917 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:38.739857 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:38.739818 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:40.102837 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:40.102799 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:40.882698 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:40.882664 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:41.005931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:41.005894 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:41.230371 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:41.230337 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:41.240597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:41.240565 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8"] Nov 01 13:04:41.240801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:41.240776 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" containerID="cri-o://8fe60b503810a4c0b1acb8077a31abc1e579938039720f4a31e39e2392d936c8" gracePeriod=30 Nov 01 13:04:45.410828 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:45.410795 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:46.382824 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.382791 4484 generic.go:356] "Generic (PLEG): container finished" podID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerID="8fe60b503810a4c0b1acb8077a31abc1e579938039720f4a31e39e2392d936c8" exitCode=0 Nov 01 13:04:46.382995 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.382873 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" event={"ID":"5a20c7db-6279-4745-b955-a3b7459b8b45","Type":"ContainerDied","Data":"8fe60b503810a4c0b1acb8077a31abc1e579938039720f4a31e39e2392d936c8"} Nov 01 13:04:46.382995 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.382953 4484 scope.go:122] "RemoveContainer" containerID="a9015bd8e959ad1d95309108eec9facd8d55a77ba40c642a666ef0be5330976c" Nov 01 13:04:46.437182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.437143 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:04:46.492312 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.492278 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 13:04:46.567381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567267 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567343 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567444 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567491 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567529 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567608 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567658 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.567780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567748 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48\") pod \"5a20c7db-6279-4745-b955-a3b7459b8b45\" (UID: \"5a20c7db-6279-4745-b955-a3b7459b8b45\") " Nov 01 13:04:46.568002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567812 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:04:46.568002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567925 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:04:46.568002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.567959 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:04:46.568164 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568046 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:04:46.568218 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568196 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:46.568378 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568291 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a20c7db-6279-4745-b955-a3b7459b8b45-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.568378 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568332 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.568378 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568342 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.568378 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568352 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.568378 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.568361 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.570123 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.570098 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:04:46.570123 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.570108 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:46.570272 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.570105 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:04:46.570339 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.570296 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48" pod "5a20c7db-6279-4745-b955-a3b7459b8b45" (UID: "5a20c7db-6279-4745-b955-a3b7459b8b45"). InnerVolumeSpecName "kube-api-access-28x48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:46.669829 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.669788 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.669829 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.669823 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.670083 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.669838 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28x48\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-kube-api-access-28x48\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:46.670083 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:46.669853 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a20c7db-6279-4745-b955-a3b7459b8b45-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:47.389795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:47.389765 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" Nov 01 13:04:47.389998 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:47.389795 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8" event={"ID":"5a20c7db-6279-4745-b955-a3b7459b8b45","Type":"ContainerDied","Data":"c12b5ae494d524f1ab3a50511c776dc3a823bc82ab6c18a8e358ac1d92561463"} Nov 01 13:04:47.389998 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:47.389864 4484 scope.go:122] "RemoveContainer" containerID="8fe60b503810a4c0b1acb8077a31abc1e579938039720f4a31e39e2392d936c8" Nov 01 13:04:47.589144 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:47.589109 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8"] Nov 01 13:04:47.592768 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:47.592744 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-wngs8"] Nov 01 13:04:48.044662 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.044587 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8"] Nov 01 13:04:48.046594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.046573 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1245f-7dbp8"] Nov 01 13:04:48.055179 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.051362 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5"] Nov 01 13:04:48.057101 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.057067 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42877763-xqrz5"] Nov 01 13:04:48.367633 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.367546 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" path="/var/lib/kubelet/pods/5a20c7db-6279-4745-b955-a3b7459b8b45/volumes" Nov 01 13:04:48.368005 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.367991 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a624144-2e97-47c7-b648-3129e4d1c821" path="/var/lib/kubelet/pods/6a624144-2e97-47c7-b648-3129e4d1c821/volumes" Nov 01 13:04:48.368389 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:48.368376 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a3201b-a468-41ae-b865-8913bf516c11" path="/var/lib/kubelet/pods/b0a3201b-a468-41ae-b865-8913bf516c11/volumes" Nov 01 13:04:49.993114 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:49.993078 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl"] Nov 01 13:04:49.993559 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:49.993341 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" podUID="27901c66-1863-4fa4-8888-ec5e7fbb2782" containerName="maas-api" containerID="cri-o://8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335" gracePeriod=30 Nov 01 13:04:50.086224 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.086192 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:50.273804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.273781 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 13:04:50.406909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.406878 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8\" (UniqueName: \"kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8\") pod \"27901c66-1863-4fa4-8888-ec5e7fbb2782\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " Nov 01 13:04:50.407086 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.407067 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\") pod \"27901c66-1863-4fa4-8888-ec5e7fbb2782\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " Nov 01 13:04:50.407152 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.407125 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca\") pod \"27901c66-1863-4fa4-8888-ec5e7fbb2782\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " Nov 01 13:04:50.407220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.407172 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs\") pod \"27901c66-1863-4fa4-8888-ec5e7fbb2782\" (UID: \"27901c66-1863-4fa4-8888-ec5e7fbb2782\") " Nov 01 13:04:50.407606 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.407574 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca" pod "27901c66-1863-4fa4-8888-ec5e7fbb2782" (UID: "27901c66-1863-4fa4-8888-ec5e7fbb2782"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:04:50.408926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.408892 4484 generic.go:356] "Generic (PLEG): container finished" podID="27901c66-1863-4fa4-8888-ec5e7fbb2782" containerID="8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335" exitCode=0 Nov 01 13:04:50.409108 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409085 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" event={"ID":"27901c66-1863-4fa4-8888-ec5e7fbb2782","Type":"ContainerDied","Data":"8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335"} Nov 01 13:04:50.409185 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409133 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" event={"ID":"27901c66-1863-4fa4-8888-ec5e7fbb2782","Type":"ContainerDied","Data":"12887eef515e23eb72654b0df0f109751b5688d8782ef8474bd47c6040435e57"} Nov 01 13:04:50.409185 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409154 4484 scope.go:122] "RemoveContainer" containerID="8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335" Nov 01 13:04:50.409342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409285 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl" Nov 01 13:04:50.409421 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409398 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8" pod "27901c66-1863-4fa4-8888-ec5e7fbb2782" (UID: "27901c66-1863-4fa4-8888-ec5e7fbb2782"). InnerVolumeSpecName "kube-api-access-grkt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:04:50.409652 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409631 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls" pod "27901c66-1863-4fa4-8888-ec5e7fbb2782" (UID: "27901c66-1863-4fa4-8888-ec5e7fbb2782"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:04:50.409789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.409769 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs" pod "27901c66-1863-4fa4-8888-ec5e7fbb2782" (UID: "27901c66-1863-4fa4-8888-ec5e7fbb2782"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:04:50.428833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.428812 4484 scope.go:122] "RemoveContainer" containerID="8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335" Nov 01 13:04:50.429095 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:04:50.429072 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335\": container with ID starting with 8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335 not found: ID does not exist" containerID="8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335" Nov 01 13:04:50.429151 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.429106 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335"} err="failed to get container status \"8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335\": rpc error: code = NotFound desc = could not find container \"8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335\": container with ID starting with 8d893e309bbf1b8acb03cc3def6f18ba09f7769546c3773babca8b61b6f16335 not found: ID does not exist" Nov 01 13:04:50.508337 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.508234 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:50.508337 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.508257 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkt8\" (UniqueName: \"kubernetes.io/projected/27901c66-1863-4fa4-8888-ec5e7fbb2782-kube-api-access-grkt8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:50.508337 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.508268 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27901c66-1863-4fa4-8888-ec5e7fbb2782-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:50.508337 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.508278 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27901c66-1863-4fa4-8888-ec5e7fbb2782-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:04:50.821882 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.821797 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl"] Nov 01 13:04:50.828114 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:50.828082 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-06fd6b-646877fdbf-mbgzl"] Nov 01 13:04:51.786432 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:51.786399 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:51.952391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:51.952356 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:52.070964 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:52.070869 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:52.190338 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:52.189554 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:52.368208 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:52.368129 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27901c66-1863-4fa4-8888-ec5e7fbb2782" path="/var/lib/kubelet/pods/27901c66-1863-4fa4-8888-ec5e7fbb2782/volumes" Nov 01 13:04:52.741233 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:52.741204 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:57.902577 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:57.902494 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:58.030630 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:58.030603 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:58.888673 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:58.888638 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:59.020792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:59.020760 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:04:59.226109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:59.226085 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv"] Nov 01 13:04:59.226369 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:59.226347 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" containerID="cri-o://8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186" gracePeriod=30 Nov 01 13:04:59.232921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:04:59.232897 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:00.251379 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:05:00.251342 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:05:03.209724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:03.209691 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:04.473665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.473641 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 13:05:04.483181 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.483149 4484 generic.go:356] "Generic (PLEG): container finished" podID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerID="8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186" exitCode=0 Nov 01 13:05:04.483372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.483197 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" Nov 01 13:05:04.483372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.483232 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" event={"ID":"9a1d5528-6aba-46ee-b3da-a7509da518b9","Type":"ContainerDied","Data":"8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186"} Nov 01 13:05:04.483372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.483323 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv" event={"ID":"9a1d5528-6aba-46ee-b3da-a7509da518b9","Type":"ContainerDied","Data":"c6f69219227d87130f2e09294a75b3e5b4dd048f02f00921ae6cc1449d539254"} Nov 01 13:05:04.483372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.483327 4484 scope.go:122] "RemoveContainer" containerID="8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186" Nov 01 13:05:04.492357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.492332 4484 scope.go:122] "RemoveContainer" containerID="8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186" Nov 01 13:05:04.492642 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:05:04.492623 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186\": container with ID starting with 8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186 not found: ID does not exist" containerID="8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186" Nov 01 13:05:04.492693 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.492652 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186"} err="failed to get container status \"8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186\": rpc error: code = NotFound desc = could not find container \"8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186\": container with ID starting with 8f79a5c55a83411097bae36304947b510449258471c8cd07e5d17f891130f186 not found: ID does not exist" Nov 01 13:05:04.658025 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.657993 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658025 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658033 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658310 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658058 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658310 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658107 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658310 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658135 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658310 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658169 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658310 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658219 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658310 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658259 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658600 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658328 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy\") pod \"9a1d5528-6aba-46ee-b3da-a7509da518b9\" (UID: \"9a1d5528-6aba-46ee-b3da-a7509da518b9\") " Nov 01 13:05:04.658600 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658423 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:05:04.658730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.658709 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1d5528-6aba-46ee-b3da-a7509da518b9-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.659063 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.659014 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:05:04.659063 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.659037 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:05:04.659244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.659059 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:05:04.659244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.659170 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:05:04.669827 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.669771 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:05:04.670061 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.670037 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:05:04.670171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.670065 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:05:04.673451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.673421 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b" pod "9a1d5528-6aba-46ee-b3da-a7509da518b9" (UID: "9a1d5528-6aba-46ee-b3da-a7509da518b9"). InnerVolumeSpecName "kube-api-access-gcj2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760184 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760217 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760227 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760235 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760244 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760253 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760261 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcj2b\" (UniqueName: \"kubernetes.io/projected/9a1d5528-6aba-46ee-b3da-a7509da518b9-kube-api-access-gcj2b\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.760637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.760277 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1d5528-6aba-46ee-b3da-a7509da518b9-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:04.897628 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.897593 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv"] Nov 01 13:05:04.902812 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:04.902788 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-2p5gv"] Nov 01 13:05:06.368438 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:06.368403 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" path="/var/lib/kubelet/pods/9a1d5528-6aba-46ee-b3da-a7509da518b9/volumes" Nov 01 13:05:08.595573 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:08.595481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:10.521871 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:10.521835 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:11.889494 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:11.889464 4484 scope.go:122] "RemoveContainer" containerID="cca226972b54678c1926dccaadee001017bb9dac6358972783ec082f3985704c" Nov 01 13:05:14.002386 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:14.002344 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:14.569172 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:14.569141 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:15.937462 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:15.937421 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:05:19.394089 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:19.394048 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:21.048094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:21.048059 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9"] Nov 01 13:05:21.093717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:21.093688 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2635cf4f-x7xl9"] Nov 01 13:05:22.367350 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:22.367316 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b03f64f-fe54-435f-a7f0-4e0533a4b776" path="/var/lib/kubelet/pods/7b03f64f-fe54-435f-a7f0-4e0533a4b776/volumes" Nov 01 13:05:24.805573 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:24.805541 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:26.043421 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:26.043383 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt"] Nov 01 13:05:26.089707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:26.089674 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a60878d2-rtvxt"] Nov 01 13:05:26.367572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:26.367490 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad86fe5-f9d4-4452-ad36-cc00679df786" path="/var/lib/kubelet/pods/2ad86fe5-f9d4-4452-ad36-cc00679df786/volumes" Nov 01 13:05:30.095079 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:30.095038 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:30.252335 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:05:30.252276 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:05:35.505654 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:35.505624 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:35.651490 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:35.651454 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:05:37.212494 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.212461 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:37.828051 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828019 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6"] Nov 01 13:05:37.828524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828505 4484 cpu_manager.go:414] "containerMap: removing container" podUID="98def0f2-4fae-4bd7-8a33-f288eef96e2d" containerName="revoke-keys" Nov 01 13:05:37.828524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828525 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98def0f2-4fae-4bd7-8a33-f288eef96e2d" containerName="revoke-keys" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828551 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828558 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828568 4484 cpu_manager.go:414] "containerMap: removing container" podUID="27901c66-1863-4fa4-8888-ec5e7fbb2782" containerName="maas-api" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828576 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27901c66-1863-4fa4-8888-ec5e7fbb2782" containerName="maas-api" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828591 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828599 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828625 4484 cpu_manager.go:414] "containerMap: removing container" podUID="99915904-ffac-435d-a0a5-da93549daca6" containerName="payload-processing" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828632 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99915904-ffac-435d-a0a5-da93549daca6" containerName="payload-processing" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828654 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" Nov 01 13:05:37.828719 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828664 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" Nov 01 13:05:37.829331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828758 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="99915904-ffac-435d-a0a5-da93549daca6" containerName="payload-processing" Nov 01 13:05:37.829331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828776 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1d5528-6aba-46ee-b3da-a7509da518b9" containerName="istio-proxy" Nov 01 13:05:37.829331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828785 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="98def0f2-4fae-4bd7-8a33-f288eef96e2d" containerName="revoke-keys" Nov 01 13:05:37.829331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828796 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" Nov 01 13:05:37.829331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828804 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="27901c66-1863-4fa4-8888-ec5e7fbb2782" containerName="maas-api" Nov 01 13:05:37.829331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.828814 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a20c7db-6279-4745-b955-a3b7459b8b45" containerName="istio-proxy" Nov 01 13:05:37.833666 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.833646 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:37.885508 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.885481 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6"] Nov 01 13:05:37.928981 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.928955 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:05:37.929104 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.928999 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:05:37.991112 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.991081 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mcx\" (UniqueName: \"kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:37.991236 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.991117 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:37.991236 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:37.991136 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.092795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.092708 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.092795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.092753 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.093061 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.092887 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mcx\" (UniqueName: \"kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.093380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.093358 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.095105 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.095083 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.146027 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.146001 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mcx\" (UniqueName: \"kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx\") pod \"maas-api-revoke-keys-e2e-inf-a-72078a-qsws6\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.152669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.152648 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:38.514818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.514794 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6"] Nov 01 13:05:38.517061 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:05:38.517020 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262c6c30_4f73_49f7_81f3_94f6b6fd4124.slice/crio-ff48be0823bae763e227595e1cd9b2ba29d6940da01190a44cc67fd9e4a7693f WatchSource:0}: Error finding container ff48be0823bae763e227595e1cd9b2ba29d6940da01190a44cc67fd9e4a7693f: Status 404 returned error can't find the container with id ff48be0823bae763e227595e1cd9b2ba29d6940da01190a44cc67fd9e4a7693f Nov 01 13:05:38.669510 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:38.669477 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" event={"ID":"262c6c30-4f73-49f7-81f3-94f6b6fd4124","Type":"ContainerStarted","Data":"ff48be0823bae763e227595e1cd9b2ba29d6940da01190a44cc67fd9e4a7693f"} Nov 01 13:05:39.676196 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:39.676164 4484 generic.go:356] "Generic (PLEG): container finished" podID="262c6c30-4f73-49f7-81f3-94f6b6fd4124" containerID="256914326ede6e818ce7da8d0373a997f8035f30896e6a46ba387fd5fd32cbb5" exitCode=0 Nov 01 13:05:39.676594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:39.676207 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" event={"ID":"262c6c30-4f73-49f7-81f3-94f6b6fd4124","Type":"ContainerDied","Data":"256914326ede6e818ce7da8d0373a997f8035f30896e6a46ba387fd5fd32cbb5"} Nov 01 13:05:40.807942 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.807916 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:40.821526 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.821500 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca\") pod \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " Nov 01 13:05:40.821649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.821541 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx\" (UniqueName: \"kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx\") pod \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " Nov 01 13:05:40.821649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.821577 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca\") pod \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\" (UID: \"262c6c30-4f73-49f7-81f3-94f6b6fd4124\") " Nov 01 13:05:40.821978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.821951 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca" pod "262c6c30-4f73-49f7-81f3-94f6b6fd4124" (UID: "262c6c30-4f73-49f7-81f3-94f6b6fd4124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:05:40.823463 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.823435 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca" pod "262c6c30-4f73-49f7-81f3-94f6b6fd4124" (UID: "262c6c30-4f73-49f7-81f3-94f6b6fd4124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:05:40.823596 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.823577 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx" pod "262c6c30-4f73-49f7-81f3-94f6b6fd4124" (UID: "262c6c30-4f73-49f7-81f3-94f6b6fd4124"). InnerVolumeSpecName "kube-api-access-z4mcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:05:40.922524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.922494 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:40.922524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.922522 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mcx\" (UniqueName: \"kubernetes.io/projected/262c6c30-4f73-49f7-81f3-94f6b6fd4124-kube-api-access-z4mcx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:40.922710 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:40.922533 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262c6c30-4f73-49f7-81f3-94f6b6fd4124-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:05:41.085514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:41.085465 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:41.689125 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:41.689095 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" Nov 01 13:05:41.689338 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:41.689092 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6" event={"ID":"262c6c30-4f73-49f7-81f3-94f6b6fd4124","Type":"ContainerDied","Data":"ff48be0823bae763e227595e1cd9b2ba29d6940da01190a44cc67fd9e4a7693f"} Nov 01 13:05:41.689338 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:41.689222 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff48be0823bae763e227595e1cd9b2ba29d6940da01190a44cc67fd9e4a7693f" Nov 01 13:05:46.626616 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:46.626570 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:48.810479 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:48.810445 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:05:49.418024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:49.417991 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n"] Nov 01 13:05:49.784838 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:49.784799 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:49.901027 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:49.900986 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:51.911394 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:51.911361 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:54.419315 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:54.419246 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerName="payload-pre-processing" containerID="cri-o://3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3" gracePeriod=25 Nov 01 13:05:56.612573 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:56.612543 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:58.344481 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:58.344440 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:58.597664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:58.597579 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:58.728739 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:58.728706 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:05:58.913133 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:58.913057 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6"] Nov 01 13:05:58.958706 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:58.958681 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-72078a-qsws6"] Nov 01 13:05:59.316801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:05:59.316756 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:00.250794 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:06:00.250757 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:06:00.367909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:00.367878 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262c6c30-4f73-49f7-81f3-94f6b6fd4124" path="/var/lib/kubelet/pods/262c6c30-4f73-49f7-81f3-94f6b6fd4124/volumes" Nov 01 13:06:02.333649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:02.333613 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:06:02.604915 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:02.604837 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:06:04.386065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:04.386029 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:05.350187 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:05.350154 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:05.497999 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:05.497966 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:05.731666 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:05.731637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:09.940595 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:09.940521 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:11.926758 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:11.926730 4484 scope.go:122] "RemoveContainer" containerID="88df0620e24ca8a8d1f1615ff7a17e3c201751f44b5ba525112df219880e7147" Nov 01 13:06:11.935360 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:11.935336 4484 scope.go:122] "RemoveContainer" containerID="5f90cc85f95590f9b3aca33d9d6a4db9ac8d7882e0d86f2b00313e6e97b1386e" Nov 01 13:06:11.943434 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:11.943411 4484 scope.go:122] "RemoveContainer" containerID="cb5ab2f09c336edab4cb6526361f3d006ec6adba75fb453c17f75acf99701bf3" Nov 01 13:06:11.951028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:11.951008 4484 scope.go:122] "RemoveContainer" containerID="343a1d6c2e2ab3730706acc09b7d050663f20fda90021532c3e2d85c8f3a89b7" Nov 01 13:06:11.958743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:11.958723 4484 scope.go:122] "RemoveContainer" containerID="4a91890dfbc18a0793f0d4a289eb00b196042ddf3d13df8d05a80687792c86a2" Nov 01 13:06:12.604068 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:12.604031 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:06:14.670806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.670784 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 13:06:14.774607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.774583 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff\" (UniqueName: \"kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff\") pod \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " Nov 01 13:06:14.774766 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.774615 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume\") pod \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\" (UID: \"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7\") " Nov 01 13:06:14.775053 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.775029 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume" pod "3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" (UID: "3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:06:14.777754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.777725 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff" pod "3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" (UID: "3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7"). InnerVolumeSpecName "kube-api-access-vz2ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:06:14.868039 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.868006 4484 generic.go:356] "Generic (PLEG): container finished" podID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerID="3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3" exitCode=0 Nov 01 13:06:14.868171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.868064 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" Nov 01 13:06:14.868171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.868097 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" event={"ID":"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7","Type":"ContainerDied","Data":"3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3"} Nov 01 13:06:14.868171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.868162 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n" event={"ID":"3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7","Type":"ContainerDied","Data":"7ee7234d083ce48c7875c42e2a09b55cf1bc8f2def9cfce80a5359c0b6a27c1f"} Nov 01 13:06:14.868286 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.868192 4484 scope.go:122] "RemoveContainer" containerID="3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3" Nov 01 13:06:14.875636 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.875615 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2ff\" (UniqueName: \"kubernetes.io/projected/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-kube-api-access-vz2ff\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:14.875775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.875639 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:14.877801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.877785 4484 scope.go:122] "RemoveContainer" containerID="3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3" Nov 01 13:06:14.878061 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:06:14.878044 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3\": container with ID starting with 3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3 not found: ID does not exist" containerID="3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3" Nov 01 13:06:14.878127 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.878068 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3"} err="failed to get container status \"3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3\": rpc error: code = NotFound desc = could not find container \"3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3\": container with ID starting with 3697dae46c416fb79675ac0534cd57c2e05bb6d77621de41a7e1f99d4b2532c3 not found: ID does not exist" Nov 01 13:06:14.881174 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:14.881154 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:15.019649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:15.019617 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n"] Nov 01 13:06:15.025358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:15.025281 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-72078a-7f7c657799-82v2n"] Nov 01 13:06:15.496946 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:15.496914 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:16.367740 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:16.367709 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" path="/var/lib/kubelet/pods/3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7/volumes" Nov 01 13:06:18.167141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:18.167107 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k"] Nov 01 13:06:18.274717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:18.274684 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw"] Nov 01 13:06:18.515251 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:18.515222 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:18.633381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:18.633349 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:20.772849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:20.772817 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:23.168163 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.168123 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" podUID="86080597-c9d7-40b3-975e-31ed344ba8d1" containerName="payload-processing" containerID="cri-o://8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1" gracePeriod=25 Nov 01 13:06:23.275374 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.275340 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerName="payload-pre-processing" containerID="cri-o://ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881" gracePeriod=25 Nov 01 13:06:23.411945 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.411923 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:06:23.464756 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.464700 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f\" (UniqueName: \"kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f\") pod \"86080597-c9d7-40b3-975e-31ed344ba8d1\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " Nov 01 13:06:23.464896 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.464815 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume\") pod \"86080597-c9d7-40b3-975e-31ed344ba8d1\" (UID: \"86080597-c9d7-40b3-975e-31ed344ba8d1\") " Nov 01 13:06:23.465146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.465123 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume" pod "86080597-c9d7-40b3-975e-31ed344ba8d1" (UID: "86080597-c9d7-40b3-975e-31ed344ba8d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:06:23.466661 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.466636 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f" pod "86080597-c9d7-40b3-975e-31ed344ba8d1" (UID: "86080597-c9d7-40b3-975e-31ed344ba8d1"). InnerVolumeSpecName "kube-api-access-jxk5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:06:23.565842 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.565787 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxk5f\" (UniqueName: \"kubernetes.io/projected/86080597-c9d7-40b3-975e-31ed344ba8d1-kube-api-access-jxk5f\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:23.565842 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.565813 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86080597-c9d7-40b3-975e-31ed344ba8d1-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:23.913608 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.913575 4484 generic.go:356] "Generic (PLEG): container finished" podID="86080597-c9d7-40b3-975e-31ed344ba8d1" containerID="8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1" exitCode=0 Nov 01 13:06:23.913773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.913620 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" Nov 01 13:06:23.913773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.913634 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" event={"ID":"86080597-c9d7-40b3-975e-31ed344ba8d1","Type":"ContainerDied","Data":"8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1"} Nov 01 13:06:23.913773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.913684 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k" event={"ID":"86080597-c9d7-40b3-975e-31ed344ba8d1","Type":"ContainerDied","Data":"b4a2a939c16465bd045bbaaf11b5f945a6440a3b757cfa7095037e498767522a"} Nov 01 13:06:23.913773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.913713 4484 scope.go:122] "RemoveContainer" containerID="8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1" Nov 01 13:06:23.923467 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.923449 4484 scope.go:122] "RemoveContainer" containerID="8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1" Nov 01 13:06:23.923756 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:06:23.923737 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1\": container with ID starting with 8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1 not found: ID does not exist" containerID="8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1" Nov 01 13:06:23.923817 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:23.923764 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1"} err="failed to get container status \"8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1\": rpc error: code = NotFound desc = could not find container \"8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1\": container with ID starting with 8b67a4f689b92dfc7a349f9bbb799dc3958eaf252169a05edbc9886ba09f55f1 not found: ID does not exist" Nov 01 13:06:24.065816 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:24.065787 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k"] Nov 01 13:06:24.111855 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:24.111822 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68322-794645fb85-4h74k"] Nov 01 13:06:24.368593 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:24.368558 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86080597-c9d7-40b3-975e-31ed344ba8d1" path="/var/lib/kubelet/pods/86080597-c9d7-40b3-975e-31ed344ba8d1/volumes" Nov 01 13:06:25.879662 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:25.879624 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:06:26.095762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:26.095727 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:30.251129 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:06:30.251088 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:06:30.794434 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:30.794401 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:32.539857 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:32.539821 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:32.782177 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:32.782137 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:32.895078 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:32.894995 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:33.480920 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:33.480883 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:34.042312 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:34.042246 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8"] Nov 01 13:06:34.088966 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:34.088935 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dca3d77b-6wgk8"] Nov 01 13:06:34.367425 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:34.367347 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9ada65-8080-43ed-aadb-d5fc97e024b5" path="/var/lib/kubelet/pods/3c9ada65-8080-43ed-aadb-d5fc97e024b5/volumes" Nov 01 13:06:35.879762 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:35.879722 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:06:37.608246 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:37.608208 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:06:38.602013 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:38.601974 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:39.402124 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:39.402088 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:39.532917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:39.532883 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:39.857009 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:39.856973 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf"] Nov 01 13:06:39.857220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:39.857194 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" containerID="cri-o://b035f905568a2042ae25d60eff2b6f4858cae9bbd475ed4e8ff0e125f2c218b5" gracePeriod=30 Nov 01 13:06:39.862354 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:39.862295 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:40.222997 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:40.222920 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:43.524513 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.524484 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:06:43.571588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.571512 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume\") pod \"1419fb27-10bb-4263-8ac5-6f2663975eb3\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " Nov 01 13:06:43.571588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.571554 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx\" (UniqueName: \"kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx\") pod \"1419fb27-10bb-4263-8ac5-6f2663975eb3\" (UID: \"1419fb27-10bb-4263-8ac5-6f2663975eb3\") " Nov 01 13:06:43.571939 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.571816 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume" pod "1419fb27-10bb-4263-8ac5-6f2663975eb3" (UID: "1419fb27-10bb-4263-8ac5-6f2663975eb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:06:43.571988 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.571966 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1419fb27-10bb-4263-8ac5-6f2663975eb3-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:43.573581 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.573558 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx" pod "1419fb27-10bb-4263-8ac5-6f2663975eb3" (UID: "1419fb27-10bb-4263-8ac5-6f2663975eb3"). InnerVolumeSpecName "kube-api-access-knwtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:06:43.672897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:43.672863 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwtx\" (UniqueName: \"kubernetes.io/projected/1419fb27-10bb-4263-8ac5-6f2663975eb3-kube-api-access-knwtx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:44.014611 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.014569 4484 generic.go:356] "Generic (PLEG): container finished" podID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerID="ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881" exitCode=0 Nov 01 13:06:44.014818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.014627 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" Nov 01 13:06:44.014818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.014654 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" event={"ID":"1419fb27-10bb-4263-8ac5-6f2663975eb3","Type":"ContainerDied","Data":"ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881"} Nov 01 13:06:44.014818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.014709 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw" event={"ID":"1419fb27-10bb-4263-8ac5-6f2663975eb3","Type":"ContainerDied","Data":"272bfef82c01bb08326515313d8c8cd755ad39cd5003ae5d9885a2e668ed221e"} Nov 01 13:06:44.014818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.014731 4484 scope.go:122] "RemoveContainer" containerID="ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881" Nov 01 13:06:44.023833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.023817 4484 scope.go:122] "RemoveContainer" containerID="ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881" Nov 01 13:06:44.024090 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:06:44.024073 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881\": container with ID starting with ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881 not found: ID does not exist" containerID="ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881" Nov 01 13:06:44.024141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.024099 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881"} err="failed to get container status \"ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881\": rpc error: code = NotFound desc = could not find container \"ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881\": container with ID starting with ccbb0347cd79e78210e2e63cdf07a2ee78ff43094b1b055426167141d2106881 not found: ID does not exist" Nov 01 13:06:44.166742 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.166707 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw"] Nov 01 13:06:44.213073 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.213038 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68322-546969c9ff-9lvgw"] Nov 01 13:06:44.367971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.367892 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" path="/var/lib/kubelet/pods/1419fb27-10bb-4263-8ac5-6f2663975eb3/volumes" Nov 01 13:06:44.417408 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:44.417375 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:45.022140 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.022108 4484 generic.go:356] "Generic (PLEG): container finished" podID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerID="b035f905568a2042ae25d60eff2b6f4858cae9bbd475ed4e8ff0e125f2c218b5" exitCode=0 Nov 01 13:06:45.022643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.022175 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" event={"ID":"e8443af5-fdb8-4072-9e1b-567609af5e6e","Type":"ContainerDied","Data":"b035f905568a2042ae25d60eff2b6f4858cae9bbd475ed4e8ff0e125f2c218b5"} Nov 01 13:06:45.022643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.022239 4484 scope.go:122] "RemoveContainer" containerID="eb6b4c103648e7bd4ccecc67e8340243ef2510d321cafc135a1d41250c6e6e29" Nov 01 13:06:45.109025 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.109001 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:06:45.186898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.186814 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.186898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.186870 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.186898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.186897 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.186917 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.186978 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187018 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187077 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187104 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187497 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187183 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs\") pod \"e8443af5-fdb8-4072-9e1b-567609af5e6e\" (UID: \"e8443af5-fdb8-4072-9e1b-567609af5e6e\") " Nov 01 13:06:45.187497 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187241 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:06:45.187620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187527 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:06:45.187620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187568 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:06:45.187620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187588 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.187620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187598 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:06:45.187789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.187613 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:06:45.189144 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.189115 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:06:45.189433 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.189413 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "kube-api-access-nxx98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:06:45.189522 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.189484 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:06:45.189681 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.189657 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo" pod "e8443af5-fdb8-4072-9e1b-567609af5e6e" (UID: "e8443af5-fdb8-4072-9e1b-567609af5e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:06:45.205973 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.205945 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm"] Nov 01 13:06:45.207831 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.207805 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62"] Nov 01 13:06:45.215293 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.215271 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c"] Nov 01 13:06:45.288601 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288571 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288601 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288598 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxx98\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-kube-api-access-nxx98\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288601 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288609 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288618 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288628 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8443af5-fdb8-4072-9e1b-567609af5e6e-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288642 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288655 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8443af5-fdb8-4072-9e1b-567609af5e6e-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.288845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.288667 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8443af5-fdb8-4072-9e1b-567609af5e6e-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:45.317628 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:45.317603 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:46.029074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:46.029042 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" Nov 01 13:06:46.029553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:46.029052 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf" event={"ID":"e8443af5-fdb8-4072-9e1b-567609af5e6e","Type":"ContainerDied","Data":"08af3cd85253183d55ad135d760348ffc7c0de0288a61416119e3f6c2604cd9a"} Nov 01 13:06:46.029553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:46.029175 4484 scope.go:122] "RemoveContainer" containerID="b035f905568a2042ae25d60eff2b6f4858cae9bbd475ed4e8ff0e125f2c218b5" Nov 01 13:06:46.181141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:46.181109 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf"] Nov 01 13:06:46.227073 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:46.227042 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b68322-openshift-default-669f549b9f-z5rtf"] Nov 01 13:06:46.374030 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:46.373956 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" path="/var/lib/kubelet/pods/e8443af5-fdb8-4072-9e1b-567609af5e6e/volumes" Nov 01 13:06:49.899411 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:49.899374 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:52.676516 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:52.676466 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:52.795614 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:52.795569 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:06:54.707092 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707055 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml"] Nov 01 13:06:54.707641 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707620 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerName="payload-pre-processing" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707642 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerName="payload-pre-processing" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707662 4484 cpu_manager.go:414] "containerMap: removing container" podUID="86080597-c9d7-40b3-975e-31ed344ba8d1" containerName="payload-processing" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707670 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86080597-c9d7-40b3-975e-31ed344ba8d1" containerName="payload-processing" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707691 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerName="payload-pre-processing" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707699 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerName="payload-pre-processing" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707708 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" Nov 01 13:06:54.707730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707715 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707742 4484 cpu_manager.go:414] "containerMap: removing container" podUID="262c6c30-4f73-49f7-81f3-94f6b6fd4124" containerName="revoke-keys" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707749 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262c6c30-4f73-49f7-81f3-94f6b6fd4124" containerName="revoke-keys" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707842 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="86080597-c9d7-40b3-975e-31ed344ba8d1" containerName="payload-processing" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707855 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf2a008-8aad-4f54-bf4c-7c6be7e1c2e7" containerName="payload-pre-processing" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707864 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707876 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707886 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="262c6c30-4f73-49f7-81f3-94f6b6fd4124" containerName="revoke-keys" Nov 01 13:06:54.708077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.707897 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1419fb27-10bb-4263-8ac5-6f2663975eb3" containerName="payload-pre-processing" Nov 01 13:06:54.712495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.712471 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.763661 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.763629 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml"] Nov 01 13:06:54.780637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.780609 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.780788 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.780762 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.780928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.780908 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z98x6\" (UniqueName: \"kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.807245 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.807218 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:06:54.807397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.807341 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:06:54.882468 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.882434 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.882645 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.882522 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.882645 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.882581 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z98x6\" (UniqueName: \"kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.883180 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.883147 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.884818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.884796 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:54.935926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:54.935890 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z98x6\" (UniqueName: \"kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6\") pod \"maas-api-revoke-sub-c804ef7c-hnhml\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:55.025986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:55.025950 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:55.389267 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:55.389192 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml"] Nov 01 13:06:55.393160 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:06:55.393132 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3896110a_ad91_4113_a6e4_f3d620c0ac22.slice/crio-e2034ee4296b4db2c064a10d8f1733e2516562ad3fa08e903ba57390b35d8a8b WatchSource:0}: Error finding container e2034ee4296b4db2c064a10d8f1733e2516562ad3fa08e903ba57390b35d8a8b: Status 404 returned error can't find the container with id e2034ee4296b4db2c064a10d8f1733e2516562ad3fa08e903ba57390b35d8a8b Nov 01 13:06:56.085216 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:56.085184 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" event={"ID":"3896110a-ad91-4113-a6e4-f3d620c0ac22","Type":"ContainerStarted","Data":"9b72f47fbb51c4acc8b80b9640f2b98148530cbb271ce4db4e2a057cc6633740"} Nov 01 13:06:56.085705 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:56.085239 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" event={"ID":"3896110a-ad91-4113-a6e4-f3d620c0ac22","Type":"ContainerStarted","Data":"e2034ee4296b4db2c064a10d8f1733e2516562ad3fa08e903ba57390b35d8a8b"} Nov 01 13:06:56.192778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:56.192719 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" podStartSLOduration=2.192702393 podStartE2EDuration="2.192702393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:06:56.191685325 +0000 UTC m=+2685.941628303" watchObservedRunningTime="2026-11-01 13:06:56.192702393 +0000 UTC m=+2685.942645361" Nov 01 13:06:57.091199 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:57.091157 4484 generic.go:356] "Generic (PLEG): container finished" podID="3896110a-ad91-4113-a6e4-f3d620c0ac22" containerID="9b72f47fbb51c4acc8b80b9640f2b98148530cbb271ce4db4e2a057cc6633740" exitCode=0 Nov 01 13:06:57.091614 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:57.091239 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" event={"ID":"3896110a-ad91-4113-a6e4-f3d620c0ac22","Type":"ContainerDied","Data":"9b72f47fbb51c4acc8b80b9640f2b98148530cbb271ce4db4e2a057cc6633740"} Nov 01 13:06:58.222829 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.222804 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:58.313565 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.313530 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6\" (UniqueName: \"kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6\") pod \"3896110a-ad91-4113-a6e4-f3d620c0ac22\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " Nov 01 13:06:58.313565 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.313569 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca\") pod \"3896110a-ad91-4113-a6e4-f3d620c0ac22\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " Nov 01 13:06:58.313818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.313603 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca\") pod \"3896110a-ad91-4113-a6e4-f3d620c0ac22\" (UID: \"3896110a-ad91-4113-a6e4-f3d620c0ac22\") " Nov 01 13:06:58.313936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.313913 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca" pod "3896110a-ad91-4113-a6e4-f3d620c0ac22" (UID: "3896110a-ad91-4113-a6e4-f3d620c0ac22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:06:58.315729 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.315700 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6" pod "3896110a-ad91-4113-a6e4-f3d620c0ac22" (UID: "3896110a-ad91-4113-a6e4-f3d620c0ac22"). InnerVolumeSpecName "kube-api-access-z98x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:06:58.315729 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.315702 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca" pod "3896110a-ad91-4113-a6e4-f3d620c0ac22" (UID: "3896110a-ad91-4113-a6e4-f3d620c0ac22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:06:58.415372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.415335 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z98x6\" (UniqueName: \"kubernetes.io/projected/3896110a-ad91-4113-a6e4-f3d620c0ac22-kube-api-access-z98x6\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:58.415372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.415366 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:58.415619 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:58.415380 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3896110a-ad91-4113-a6e4-f3d620c0ac22-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:06:59.102611 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:59.102580 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" Nov 01 13:06:59.102825 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:59.102575 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml" event={"ID":"3896110a-ad91-4113-a6e4-f3d620c0ac22","Type":"ContainerDied","Data":"e2034ee4296b4db2c064a10d8f1733e2516562ad3fa08e903ba57390b35d8a8b"} Nov 01 13:06:59.102825 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:06:59.102708 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2034ee4296b4db2c064a10d8f1733e2516562ad3fa08e903ba57390b35d8a8b" Nov 01 13:07:00.225620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.225576 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" containerID="cri-o://b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66" gracePeriod=45 Nov 01 13:07:00.226418 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.226237 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" containerID="cri-o://edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad" gracePeriod=45 Nov 01 13:07:00.233159 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.233128 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" containerID="cri-o://594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b" gracePeriod=45 Nov 01 13:07:00.256847 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:07:00.256813 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:07:00.256954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.256855 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused" Nov 01 13:07:00.495120 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.495097 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 13:07:00.505931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.505905 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 13:07:00.520059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.520039 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 13:07:00.635927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.635889 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir\") pod \"83b07113-7770-4b13-ad2b-3c81057f4523\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " Nov 01 13:07:00.636100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.635945 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache\") pod \"5bdffd62-0423-421a-a682-2165a6e00409\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " Nov 01 13:07:00.636100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.635983 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache\") pod \"83b07113-7770-4b13-ad2b-3c81057f4523\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " Nov 01 13:07:00.636100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636083 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache\") pod \"c60a3282-fa6e-4935-a053-196324513375\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636116 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx\" (UniqueName: \"kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx\") pod \"83b07113-7770-4b13-ad2b-3c81057f4523\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636155 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir\") pod \"5bdffd62-0423-421a-a682-2165a6e00409\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636199 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home\") pod \"c60a3282-fa6e-4935-a053-196324513375\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636207 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir" pod "83b07113-7770-4b13-ad2b-3c81057f4523" (UID: "83b07113-7770-4b13-ad2b-3c81057f4523"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636230 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs\") pod \"c60a3282-fa6e-4935-a053-196324513375\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636219 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache" pod "5bdffd62-0423-421a-a682-2165a6e00409" (UID: "5bdffd62-0423-421a-a682-2165a6e00409"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636286 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng\" (UniqueName: \"kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng\") pod \"c60a3282-fa6e-4935-a053-196324513375\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636342 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache" pod "83b07113-7770-4b13-ad2b-3c81057f4523" (UID: "83b07113-7770-4b13-ad2b-3c81057f4523"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.636364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636353 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm\") pod \"83b07113-7770-4b13-ad2b-3c81057f4523\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636388 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache" pod "c60a3282-fa6e-4935-a053-196324513375" (UID: "c60a3282-fa6e-4935-a053-196324513375"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636464 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home" pod "c60a3282-fa6e-4935-a053-196324513375" (UID: "c60a3282-fa6e-4935-a053-196324513375"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636476 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir" pod "5bdffd62-0423-421a-a682-2165a6e00409" (UID: "5bdffd62-0423-421a-a682-2165a6e00409"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636417 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm\") pod \"c60a3282-fa6e-4935-a053-196324513375\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636586 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz\" (UniqueName: \"kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz\") pod \"5bdffd62-0423-421a-a682-2165a6e00409\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636624 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home\") pod \"5bdffd62-0423-421a-a682-2165a6e00409\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636697 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs\" (UniqueName: \"kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs\") pod \"83b07113-7770-4b13-ad2b-3c81057f4523\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636727 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs\") pod \"5bdffd62-0423-421a-a682-2165a6e00409\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636765 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir\") pod \"c60a3282-fa6e-4935-a053-196324513375\" (UID: \"c60a3282-fa6e-4935-a053-196324513375\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636814 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home\") pod \"83b07113-7770-4b13-ad2b-3c81057f4523\" (UID: \"83b07113-7770-4b13-ad2b-3c81057f4523\") " Nov 01 13:07:00.636921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.636840 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm\") pod \"5bdffd62-0423-421a-a682-2165a6e00409\" (UID: \"5bdffd62-0423-421a-a682-2165a6e00409\") " Nov 01 13:07:00.637548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637216 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.637548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637237 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.637548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637251 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.637548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637265 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.637548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637278 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.637548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637291 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.637859 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.637828 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir" pod "c60a3282-fa6e-4935-a053-196324513375" (UID: "c60a3282-fa6e-4935-a053-196324513375"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.638171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.638142 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home" pod "5bdffd62-0423-421a-a682-2165a6e00409" (UID: "5bdffd62-0423-421a-a682-2165a6e00409"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.638171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.638148 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home" pod "83b07113-7770-4b13-ad2b-3c81057f4523" (UID: "83b07113-7770-4b13-ad2b-3c81057f4523"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.639165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.639140 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm" pod "83b07113-7770-4b13-ad2b-3c81057f4523" (UID: "83b07113-7770-4b13-ad2b-3c81057f4523"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.639256 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.639166 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng" pod "c60a3282-fa6e-4935-a053-196324513375" (UID: "c60a3282-fa6e-4935-a053-196324513375"). InnerVolumeSpecName "kube-api-access-xgkng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:07:00.640061 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.640021 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm" pod "5bdffd62-0423-421a-a682-2165a6e00409" (UID: "5bdffd62-0423-421a-a682-2165a6e00409"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.640182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.640072 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs" pod "c60a3282-fa6e-4935-a053-196324513375" (UID: "c60a3282-fa6e-4935-a053-196324513375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:07:00.640182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.640086 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm" pod "c60a3282-fa6e-4935-a053-196324513375" (UID: "c60a3282-fa6e-4935-a053-196324513375"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:07:00.640444 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.640421 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx" pod "83b07113-7770-4b13-ad2b-3c81057f4523" (UID: "83b07113-7770-4b13-ad2b-3c81057f4523"). InnerVolumeSpecName "kube-api-access-f2mbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:07:00.640504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.640474 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz" pod "5bdffd62-0423-421a-a682-2165a6e00409" (UID: "5bdffd62-0423-421a-a682-2165a6e00409"). InnerVolumeSpecName "kube-api-access-w8kzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:07:00.641003 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.640980 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs" pod "83b07113-7770-4b13-ad2b-3c81057f4523" (UID: "83b07113-7770-4b13-ad2b-3c81057f4523"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:07:00.641103 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.641082 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs" pod "5bdffd62-0423-421a-a682-2165a6e00409" (UID: "5bdffd62-0423-421a-a682-2165a6e00409"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739180 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mbx\" (UniqueName: \"kubernetes.io/projected/83b07113-7770-4b13-ad2b-3c81057f4523-kube-api-access-f2mbx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739207 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60a3282-fa6e-4935-a053-196324513375-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739217 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgkng\" (UniqueName: \"kubernetes.io/projected/c60a3282-fa6e-4935-a053-196324513375-kube-api-access-xgkng\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739227 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739236 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739244 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8kzz\" (UniqueName: \"kubernetes.io/projected/5bdffd62-0423-421a-a682-2165a6e00409-kube-api-access-w8kzz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739255 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739264 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b07113-7770-4b13-ad2b-3c81057f4523-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739273 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdffd62-0423-421a-a682-2165a6e00409-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739281 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60a3282-fa6e-4935-a053-196324513375-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739289 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b07113-7770-4b13-ad2b-3c81057f4523-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:00.739678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:00.739296 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdffd62-0423-421a-a682-2165a6e00409-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:01.113504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.113399 4484 generic.go:356] "Generic (PLEG): container finished" podID="83b07113-7770-4b13-ad2b-3c81057f4523" containerID="594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b" exitCode=0 Nov 01 13:07:01.113504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.113444 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" Nov 01 13:07:01.113504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.113494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" event={"ID":"83b07113-7770-4b13-ad2b-3c81057f4523","Type":"ContainerDied","Data":"594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b"} Nov 01 13:07:01.113809 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.113543 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c" event={"ID":"83b07113-7770-4b13-ad2b-3c81057f4523","Type":"ContainerDied","Data":"e509e2e5cafe88187207cd9608c9826681d8361aa5f12d61834ddfbcc2270590"} Nov 01 13:07:01.113809 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.113562 4484 scope.go:122] "RemoveContainer" containerID="594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b" Nov 01 13:07:01.114822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.114798 4484 generic.go:356] "Generic (PLEG): container finished" podID="c60a3282-fa6e-4935-a053-196324513375" containerID="edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad" exitCode=0 Nov 01 13:07:01.114964 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.114835 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" Nov 01 13:07:01.114964 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.114879 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" event={"ID":"c60a3282-fa6e-4935-a053-196324513375","Type":"ContainerDied","Data":"edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad"} Nov 01 13:07:01.114964 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.114918 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62" event={"ID":"c60a3282-fa6e-4935-a053-196324513375","Type":"ContainerDied","Data":"218b424a954acf736e5565a050bc6e69568a3dc2a5816d1ac21082c7a3e3bc26"} Nov 01 13:07:01.116398 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.116378 4484 generic.go:356] "Generic (PLEG): container finished" podID="5bdffd62-0423-421a-a682-2165a6e00409" containerID="b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66" exitCode=0 Nov 01 13:07:01.116504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.116428 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" Nov 01 13:07:01.116504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.116486 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" event={"ID":"5bdffd62-0423-421a-a682-2165a6e00409","Type":"ContainerDied","Data":"b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66"} Nov 01 13:07:01.116576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.116526 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm" event={"ID":"5bdffd62-0423-421a-a682-2165a6e00409","Type":"ContainerDied","Data":"3ce26426d81d7ede6b6301ef09e741165b981bf54433aebdfea04accff36b466"} Nov 01 13:07:01.123035 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.123016 4484 scope.go:122] "RemoveContainer" containerID="594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b" Nov 01 13:07:01.123506 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:07:01.123389 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b\": container with ID starting with 594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b not found: ID does not exist" containerID="594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b" Nov 01 13:07:01.123506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.123418 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b"} err="failed to get container status \"594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b\": rpc error: code = NotFound desc = could not find container \"594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b\": container with ID starting with 594fdd05592b975a60a5fcfb43dc86a8c9a01479ee8e1b19f1124edef988886b not found: ID does not exist" Nov 01 13:07:01.123506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.123435 4484 scope.go:122] "RemoveContainer" containerID="edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad" Nov 01 13:07:01.132314 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.132283 4484 scope.go:122] "RemoveContainer" containerID="edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad" Nov 01 13:07:01.132546 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:07:01.132528 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad\": container with ID starting with edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad not found: ID does not exist" containerID="edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad" Nov 01 13:07:01.132597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.132554 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad"} err="failed to get container status \"edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad\": rpc error: code = NotFound desc = could not find container \"edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad\": container with ID starting with edb7c35036176cfdcf65921e420e78fb034ab4e36723bdbb40ad8141dcf491ad not found: ID does not exist" Nov 01 13:07:01.132597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.132570 4484 scope.go:122] "RemoveContainer" containerID="b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66" Nov 01 13:07:01.140123 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.140105 4484 scope.go:122] "RemoveContainer" containerID="b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66" Nov 01 13:07:01.140398 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:07:01.140381 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66\": container with ID starting with b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66 not found: ID does not exist" containerID="b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66" Nov 01 13:07:01.140447 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.140405 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66"} err="failed to get container status \"b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66\": rpc error: code = NotFound desc = could not find container \"b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66\": container with ID starting with b4939e1f2440a8ea74d7ca7428ab64f529a04ae08cc791222b1db23dd39e6c66 not found: ID does not exist" Nov 01 13:07:01.268399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.268362 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62"] Nov 01 13:07:01.314385 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.314356 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-w6-f7427d-kserve-85d5b8778f-wxm62"] Nov 01 13:07:01.375826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.375746 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c"] Nov 01 13:07:01.379644 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.379620 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/simulator-w6-f7427d-kserve-5c7b84f587-l7x7c"] Nov 01 13:07:01.480539 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.480505 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm"] Nov 01 13:07:01.484338 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:01.484312 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/distinct-2-w6-f7427d-kserve-8bf685fcd-t2qhm"] Nov 01 13:07:02.367972 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:02.367928 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdffd62-0423-421a-a682-2165a6e00409" path="/var/lib/kubelet/pods/5bdffd62-0423-421a-a682-2165a6e00409/volumes" Nov 01 13:07:02.368543 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:02.368523 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" path="/var/lib/kubelet/pods/83b07113-7770-4b13-ad2b-3c81057f4523/volumes" Nov 01 13:07:02.369007 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:02.368986 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60a3282-fa6e-4935-a053-196324513375" path="/var/lib/kubelet/pods/c60a3282-fa6e-4935-a053-196324513375/volumes" Nov 01 13:07:10.368217 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.368182 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg"] Nov 01 13:07:10.368698 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.368408 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" podUID="5a4ab1d3-6b74-483a-9602-03100e07b77b" containerName="maas-api" containerID="cri-o://3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51" gracePeriod=30 Nov 01 13:07:10.462991 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.462956 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:10.643368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.643342 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 13:07:10.727030 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.727004 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca\") pod \"5a4ab1d3-6b74-483a-9602-03100e07b77b\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " Nov 01 13:07:10.727214 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.727091 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs\") pod \"5a4ab1d3-6b74-483a-9602-03100e07b77b\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " Nov 01 13:07:10.727214 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.727161 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm\" (UniqueName: \"kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm\") pod \"5a4ab1d3-6b74-483a-9602-03100e07b77b\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " Nov 01 13:07:10.727214 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.727200 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\") pod \"5a4ab1d3-6b74-483a-9602-03100e07b77b\" (UID: \"5a4ab1d3-6b74-483a-9602-03100e07b77b\") " Nov 01 13:07:10.727432 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.727408 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca" pod "5a4ab1d3-6b74-483a-9602-03100e07b77b" (UID: "5a4ab1d3-6b74-483a-9602-03100e07b77b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:07:10.727599 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.727570 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4ab1d3-6b74-483a-9602-03100e07b77b-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:10.731736 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.731713 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm" pod "5a4ab1d3-6b74-483a-9602-03100e07b77b" (UID: "5a4ab1d3-6b74-483a-9602-03100e07b77b"). InnerVolumeSpecName "kube-api-access-xpsvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:07:10.731854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.731802 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs" pod "5a4ab1d3-6b74-483a-9602-03100e07b77b" (UID: "5a4ab1d3-6b74-483a-9602-03100e07b77b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:07:10.740434 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.740400 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls" pod "5a4ab1d3-6b74-483a-9602-03100e07b77b" (UID: "5a4ab1d3-6b74-483a-9602-03100e07b77b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:07:10.828163 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.828122 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:10.828163 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.828158 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpsvm\" (UniqueName: \"kubernetes.io/projected/5a4ab1d3-6b74-483a-9602-03100e07b77b-kube-api-access-xpsvm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:10.828394 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:10.828173 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4ab1d3-6b74-483a-9602-03100e07b77b-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:11.177185 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.177144 4484 generic.go:356] "Generic (PLEG): container finished" podID="5a4ab1d3-6b74-483a-9602-03100e07b77b" containerID="3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51" exitCode=0 Nov 01 13:07:11.177414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.177192 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" Nov 01 13:07:11.177414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.177221 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" event={"ID":"5a4ab1d3-6b74-483a-9602-03100e07b77b","Type":"ContainerDied","Data":"3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51"} Nov 01 13:07:11.177414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.177265 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg" event={"ID":"5a4ab1d3-6b74-483a-9602-03100e07b77b","Type":"ContainerDied","Data":"5d4606e636846b885bee518082c7813355215a84ce27724bbe0df3dd0425b313"} Nov 01 13:07:11.177414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.177290 4484 scope.go:122] "RemoveContainer" containerID="3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51" Nov 01 13:07:11.187233 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.187214 4484 scope.go:122] "RemoveContainer" containerID="3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51" Nov 01 13:07:11.187536 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:07:11.187504 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51\": container with ID starting with 3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51 not found: ID does not exist" containerID="3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51" Nov 01 13:07:11.187607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.187545 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51"} err="failed to get container status \"3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51\": rpc error: code = NotFound desc = could not find container \"3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51\": container with ID starting with 3f3213f0cd393e472b0d6da7fdd5bed9f3f6a6f1d8997b9409e6f53a57549a51 not found: ID does not exist" Nov 01 13:07:11.333215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.333178 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg"] Nov 01 13:07:11.335786 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.335763 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7427d-7dbf9d74d5-2lgvg"] Nov 01 13:07:11.430902 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.430827 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:07:11.999788 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:11.999762 4484 scope.go:122] "RemoveContainer" containerID="022b20c0b7feb58c30e95e5a7b024d606c9d7ab53b97a1e0d36a746df2e625c5" Nov 01 13:07:12.010160 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:12.010140 4484 scope.go:122] "RemoveContainer" containerID="4f77ca7c4651cd07b279e2294679551fe35a44f353db9a4f53e6df481ce95e20" Nov 01 13:07:12.024574 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:12.022493 4484 scope.go:122] "RemoveContainer" containerID="997d919a096d23008c6535535189a4a06e2c25bec2732b4109404bfc98510084" Nov 01 13:07:12.171699 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:12.171661 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:12.367706 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:12.367632 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4ab1d3-6b74-483a-9602-03100e07b77b" path="/var/lib/kubelet/pods/5a4ab1d3-6b74-483a-9602-03100e07b77b/volumes" Nov 01 13:07:12.482141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:12.482104 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:13.117680 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:13.117646 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:15.219954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:15.219922 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:19.083088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:19.083054 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:19.210173 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:19.210140 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:19.430324 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:19.430229 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:21.091089 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:21.091054 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:21.828844 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:21.828800 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:21.948717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:21.948681 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:23.612822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:23.612785 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:26.574129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574092 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57458d666c-mzdpj"] Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574467 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574478 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574491 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574496 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574515 4484 cpu_manager.go:414] "containerMap: removing container" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574520 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574526 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574531 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8443af5-fdb8-4072-9e1b-567609af5e6e" containerName="istio-proxy" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574542 4484 cpu_manager.go:414] "containerMap: removing container" podUID="5a4ab1d3-6b74-483a-9602-03100e07b77b" containerName="maas-api" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574546 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4ab1d3-6b74-483a-9602-03100e07b77b" containerName="maas-api" Nov 01 13:07:26.574549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574555 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3896110a-ad91-4113-a6e4-f3d620c0ac22" containerName="revoke-keys" Nov 01 13:07:26.574895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574560 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3896110a-ad91-4113-a6e4-f3d620c0ac22" containerName="revoke-keys" Nov 01 13:07:26.574895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574610 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3896110a-ad91-4113-a6e4-f3d620c0ac22" containerName="revoke-keys" Nov 01 13:07:26.574895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574622 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdffd62-0423-421a-a682-2165a6e00409" containerName="main" Nov 01 13:07:26.574895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574628 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60a3282-fa6e-4935-a053-196324513375" containerName="main" Nov 01 13:07:26.574895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574637 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b07113-7770-4b13-ad2b-3c81057f4523" containerName="main" Nov 01 13:07:26.574895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.574642 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4ab1d3-6b74-483a-9602-03100e07b77b" containerName="maas-api" Nov 01 13:07:26.579789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.579766 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.669174 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.669121 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qvv\" (UniqueName: \"kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.669174 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.669171 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.669430 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.669322 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.674496 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.674456 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 13:07:26.674496 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.674463 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 13:07:26.674729 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.674587 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q9pth\"" Nov 01 13:07:26.675024 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.675006 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 13:07:26.677591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.677569 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57458d666c-mzdpj"] Nov 01 13:07:26.769861 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.769829 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qvv\" (UniqueName: \"kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.769861 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.769867 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.770085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.769894 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.772295 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.772266 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.772461 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.772371 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.823664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.823633 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qvv\" (UniqueName: \"kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv\") pod \"maas-controller-57458d666c-mzdpj\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.891886 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.891816 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:26.938751 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:26.938727 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:07:27.302086 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:27.302054 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57458d666c-mzdpj"] Nov 01 13:07:27.305780 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:07:27.305749 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89cfae1_12f0_4d13_b8ad_3441e2a89997.slice/crio-9f21145748ef0f649fd3a77757de0e2c0ab9bade16986b5498b3e917eb3c6f83 WatchSource:0}: Error finding container 9f21145748ef0f649fd3a77757de0e2c0ab9bade16986b5498b3e917eb3c6f83: Status 404 returned error can't find the container with id 9f21145748ef0f649fd3a77757de0e2c0ab9bade16986b5498b3e917eb3c6f83 Nov 01 13:07:27.307818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:27.307783 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 13:07:28.291966 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:28.291925 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mzdpj" event={"ID":"c89cfae1-12f0-4d13-b8ad-3441e2a89997","Type":"ContainerStarted","Data":"9f21145748ef0f649fd3a77757de0e2c0ab9bade16986b5498b3e917eb3c6f83"} Nov 01 13:07:29.092323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:29.091352 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:07:30.256538 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:07:30.256497 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:07:31.321159 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:31.321062 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mzdpj" event={"ID":"c89cfae1-12f0-4d13-b8ad-3441e2a89997","Type":"ContainerStarted","Data":"87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8"} Nov 01 13:07:31.321604 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:31.321266 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:31.442014 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:31.441963 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57458d666c-mzdpj" podStartSLOduration=1.722440382 podStartE2EDuration="5.441950349s" totalImagesPullingTime="3.719509967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:07:31.440889471 +0000 UTC m=+2721.190832449" watchObservedRunningTime="2026-11-01 13:07:31.441950349 +0000 UTC m=+2721.191893343" Nov 01 13:07:42.328863 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:42.328831 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:07:51.091832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.091795 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb"] Nov 01 13:07:51.095365 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.095343 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.197359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.197326 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:07:51.197359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.197329 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:07:51.197595 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.197412 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:07:51.197595 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.197496 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:07:51.199289 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.199266 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb"] Nov 01 13:07:51.199669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.199652 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.199721 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.199682 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpvq\" (UniqueName: \"kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.199822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.199806 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.300578 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.300537 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpvq\" (UniqueName: \"kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.300778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.300640 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.300778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.300709 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.301259 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.301234 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.303083 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.303060 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.353252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.353185 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpvq\" (UniqueName: \"kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq\") pod \"maas-api-revoke-sub-85237e45-2xqkb\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.408139 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.408115 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:51.770094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:51.770062 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb"] Nov 01 13:07:51.773180 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:07:51.773150 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89bc765_5939_4636_9c74_e12f4b7588be.slice/crio-bbd62727750105228362948ed25583381335d29281659318ac3115289b4ae697 WatchSource:0}: Error finding container bbd62727750105228362948ed25583381335d29281659318ac3115289b4ae697: Status 404 returned error can't find the container with id bbd62727750105228362948ed25583381335d29281659318ac3115289b4ae697 Nov 01 13:07:52.414649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:52.414617 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" event={"ID":"e89bc765-5939-4636-9c74-e12f4b7588be","Type":"ContainerStarted","Data":"218f6461669831af35a35c9975373b9781e5f134bbf13f5cb6660089da42411b"} Nov 01 13:07:52.415018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:52.414669 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" event={"ID":"e89bc765-5939-4636-9c74-e12f4b7588be","Type":"ContainerStarted","Data":"bbd62727750105228362948ed25583381335d29281659318ac3115289b4ae697"} Nov 01 13:07:52.520122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:52.520075 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" podStartSLOduration=1.520061168 podStartE2EDuration="1.520061168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:07:52.518585179 +0000 UTC m=+2742.268528158" watchObservedRunningTime="2026-11-01 13:07:52.520061168 +0000 UTC m=+2742.270004145" Nov 01 13:07:53.420707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:53.420671 4484 generic.go:356] "Generic (PLEG): container finished" podID="e89bc765-5939-4636-9c74-e12f4b7588be" containerID="218f6461669831af35a35c9975373b9781e5f134bbf13f5cb6660089da42411b" exitCode=0 Nov 01 13:07:53.421162 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:53.420754 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" event={"ID":"e89bc765-5939-4636-9c74-e12f4b7588be","Type":"ContainerDied","Data":"218f6461669831af35a35c9975373b9781e5f134bbf13f5cb6660089da42411b"} Nov 01 13:07:54.554316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.554285 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:54.631900 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.631870 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca\") pod \"e89bc765-5939-4636-9c74-e12f4b7588be\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " Nov 01 13:07:54.632077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.631922 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca\") pod \"e89bc765-5939-4636-9c74-e12f4b7588be\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " Nov 01 13:07:54.632077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.631978 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq\" (UniqueName: \"kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq\") pod \"e89bc765-5939-4636-9c74-e12f4b7588be\" (UID: \"e89bc765-5939-4636-9c74-e12f4b7588be\") " Nov 01 13:07:54.632342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.632290 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca" pod "e89bc765-5939-4636-9c74-e12f4b7588be" (UID: "e89bc765-5939-4636-9c74-e12f4b7588be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:07:54.634049 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.634028 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca" pod "e89bc765-5939-4636-9c74-e12f4b7588be" (UID: "e89bc765-5939-4636-9c74-e12f4b7588be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:07:54.634163 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.634148 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq" pod "e89bc765-5939-4636-9c74-e12f4b7588be" (UID: "e89bc765-5939-4636-9c74-e12f4b7588be"). InnerVolumeSpecName "kube-api-access-ljpvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:07:54.733161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.733138 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:54.733332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.733165 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89bc765-5939-4636-9c74-e12f4b7588be-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:54.733332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:54.733177 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljpvq\" (UniqueName: \"kubernetes.io/projected/e89bc765-5939-4636-9c74-e12f4b7588be-kube-api-access-ljpvq\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:07:55.253538 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:55.253502 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:07:55.430897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:55.430868 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" Nov 01 13:07:55.431054 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:55.430901 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb" event={"ID":"e89bc765-5939-4636-9c74-e12f4b7588be","Type":"ContainerDied","Data":"bbd62727750105228362948ed25583381335d29281659318ac3115289b4ae697"} Nov 01 13:07:55.431054 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:07:55.430972 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd62727750105228362948ed25583381335d29281659318ac3115289b4ae697" Nov 01 13:08:00.250792 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:08:00.250757 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:08:03.371978 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.371931 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:08:03.455172 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.455142 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57458d666c-mzdpj"] Nov 01 13:08:03.455380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.455345 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-57458d666c-mzdpj" podUID="c89cfae1-12f0-4d13-b8ad-3441e2a89997" containerName="manager" containerID="cri-o://87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8" gracePeriod=30 Nov 01 13:08:03.699148 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.699126 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:08:03.815681 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.815654 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert\") pod \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " Nov 01 13:08:03.815833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.815746 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv\" (UniqueName: \"kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv\") pod \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " Nov 01 13:08:03.815833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.815786 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs\") pod \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\" (UID: \"c89cfae1-12f0-4d13-b8ad-3441e2a89997\") " Nov 01 13:08:03.817668 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.817643 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert" pod "c89cfae1-12f0-4d13-b8ad-3441e2a89997" (UID: "c89cfae1-12f0-4d13-b8ad-3441e2a89997"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:08:03.817769 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.817738 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv" pod "c89cfae1-12f0-4d13-b8ad-3441e2a89997" (UID: "c89cfae1-12f0-4d13-b8ad-3441e2a89997"). InnerVolumeSpecName "kube-api-access-47qvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:08:03.817769 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.817740 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs" pod "c89cfae1-12f0-4d13-b8ad-3441e2a89997" (UID: "c89cfae1-12f0-4d13-b8ad-3441e2a89997"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:08:03.916659 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.916617 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qvv\" (UniqueName: \"kubernetes.io/projected/c89cfae1-12f0-4d13-b8ad-3441e2a89997-kube-api-access-47qvv\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:08:03.916659 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.916660 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:08:03.916849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:03.916673 4484 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c89cfae1-12f0-4d13-b8ad-3441e2a89997-webhook-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:08:04.472042 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.472008 4484 generic.go:356] "Generic (PLEG): container finished" podID="c89cfae1-12f0-4d13-b8ad-3441e2a89997" containerID="87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8" exitCode=0 Nov 01 13:08:04.472542 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.472046 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mzdpj" Nov 01 13:08:04.472542 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.472083 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mzdpj" event={"ID":"c89cfae1-12f0-4d13-b8ad-3441e2a89997","Type":"ContainerDied","Data":"87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8"} Nov 01 13:08:04.472542 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.472133 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mzdpj" event={"ID":"c89cfae1-12f0-4d13-b8ad-3441e2a89997","Type":"ContainerDied","Data":"9f21145748ef0f649fd3a77757de0e2c0ab9bade16986b5498b3e917eb3c6f83"} Nov 01 13:08:04.472542 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.472165 4484 scope.go:122] "RemoveContainer" containerID="87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8" Nov 01 13:08:04.480667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.480647 4484 scope.go:122] "RemoveContainer" containerID="87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8" Nov 01 13:08:04.480949 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:08:04.480930 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8\": container with ID starting with 87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8 not found: ID does not exist" containerID="87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8" Nov 01 13:08:04.481034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.480958 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8"} err="failed to get container status \"87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8\": rpc error: code = NotFound desc = could not find container \"87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8\": container with ID starting with 87ce53bcdda18c4521ce9fa8f728dac9163236ea178ae7927e8719988a6765b8 not found: ID does not exist" Nov 01 13:08:04.625500 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.625424 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57458d666c-mzdpj"] Nov 01 13:08:04.628154 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:04.628130 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57458d666c-mzdpj"] Nov 01 13:08:05.894011 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:05.893982 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:08:06.054163 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054132 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57458d666c-fg9qf"] Nov 01 13:08:06.054518 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054504 4484 cpu_manager.go:414] "containerMap: removing container" podUID="c89cfae1-12f0-4d13-b8ad-3441e2a89997" containerName="manager" Nov 01 13:08:06.054572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054520 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89cfae1-12f0-4d13-b8ad-3441e2a89997" containerName="manager" Nov 01 13:08:06.054572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054530 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e89bc765-5939-4636-9c74-e12f4b7588be" containerName="revoke-keys" Nov 01 13:08:06.054572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054535 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89bc765-5939-4636-9c74-e12f4b7588be" containerName="revoke-keys" Nov 01 13:08:06.054677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054603 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89cfae1-12f0-4d13-b8ad-3441e2a89997" containerName="manager" Nov 01 13:08:06.054677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.054611 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89bc765-5939-4636-9c74-e12f4b7588be" containerName="revoke-keys" Nov 01 13:08:06.059851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.059833 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.136030 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.135998 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4584db68-8ecc-4eff-9658-108b668d04d6-metrics-certs\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.136184 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.136047 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58tp\" (UniqueName: \"kubernetes.io/projected/4584db68-8ecc-4eff-9658-108b668d04d6-kube-api-access-w58tp\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.136184 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.136080 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4584db68-8ecc-4eff-9658-108b668d04d6-webhook-cert\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.154284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.154183 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 13:08:06.154284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.154236 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q9pth\"" Nov 01 13:08:06.154284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.154269 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 13:08:06.154729 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.154704 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 13:08:06.156380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.156358 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57458d666c-fg9qf"] Nov 01 13:08:06.236807 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.236773 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4584db68-8ecc-4eff-9658-108b668d04d6-metrics-certs\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.236976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.236821 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58tp\" (UniqueName: \"kubernetes.io/projected/4584db68-8ecc-4eff-9658-108b668d04d6-kube-api-access-w58tp\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.236976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.236855 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4584db68-8ecc-4eff-9658-108b668d04d6-webhook-cert\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.239264 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.239241 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4584db68-8ecc-4eff-9658-108b668d04d6-metrics-certs\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.239397 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.239323 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4584db68-8ecc-4eff-9658-108b668d04d6-webhook-cert\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.289993 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.289961 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58tp\" (UniqueName: \"kubernetes.io/projected/4584db68-8ecc-4eff-9658-108b668d04d6-kube-api-access-w58tp\") pod \"maas-controller-57458d666c-fg9qf\" (UID: \"4584db68-8ecc-4eff-9658-108b668d04d6\") " pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.372515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.372490 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:06.373164 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.373140 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89cfae1-12f0-4d13-b8ad-3441e2a89997" path="/var/lib/kubelet/pods/c89cfae1-12f0-4d13-b8ad-3441e2a89997/volumes" Nov 01 13:08:06.735669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:06.735628 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57458d666c-fg9qf"] Nov 01 13:08:06.738671 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:08:06.738640 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4584db68_8ecc_4eff_9658_108b668d04d6.slice/crio-11bc2c10733cdd8ead071c072ceeedc7b35c34c2607e71524435f42c7f8fb967 WatchSource:0}: Error finding container 11bc2c10733cdd8ead071c072ceeedc7b35c34c2607e71524435f42c7f8fb967: Status 404 returned error can't find the container with id 11bc2c10733cdd8ead071c072ceeedc7b35c34c2607e71524435f42c7f8fb967 Nov 01 13:08:07.488066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:07.488030 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-fg9qf" event={"ID":"4584db68-8ecc-4eff-9658-108b668d04d6","Type":"ContainerStarted","Data":"11bc2c10733cdd8ead071c072ceeedc7b35c34c2607e71524435f42c7f8fb967"} Nov 01 13:08:08.493913 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:08.493878 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-fg9qf" event={"ID":"4584db68-8ecc-4eff-9658-108b668d04d6","Type":"ContainerStarted","Data":"f7afabee4f0f46ff52d5f20b7926df78271173d9febdc21cd076ba5d6210d5ed"} Nov 01 13:08:08.494293 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:08.494008 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:08.601635 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:08.601542 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57458d666c-fg9qf" podStartSLOduration=1.4000835010000001 podStartE2EDuration="2.601526589s" totalImagesPullingTime="1.201443088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:08:08.600001616 +0000 UTC m=+2758.349944597" watchObservedRunningTime="2026-11-01 13:08:08.601526589 +0000 UTC m=+2758.351469567" Nov 01 13:08:12.074856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:12.074827 4484 scope.go:122] "RemoveContainer" containerID="aacd41d5f97a0e04460d356a355b957811e930438802144a772589b9905c14da" Nov 01 13:08:12.082801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:12.082785 4484 scope.go:122] "RemoveContainer" containerID="46bd0bcaeb1f13430027d75e9dfdbbf5a4bb91736b3a23a05fcbf65504445a6b" Nov 01 13:08:15.570289 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:15.570235 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:08:19.500866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:19.500836 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57458d666c-fg9qf" Nov 01 13:08:24.321021 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:24.320978 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:08:30.254419 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:08:30.254378 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:08:48.932219 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:48.932185 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:08:57.414746 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.414709 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg"] Nov 01 13:08:57.418521 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.418501 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.471676 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.471640 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg"] Nov 01 13:08:57.493910 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.493873 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.494094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.493919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxnw\" (UniqueName: \"kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.494094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.494039 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.515621 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.515593 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:08:57.515621 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.515604 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:08:57.515788 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.515623 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:08:57.515788 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.515614 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:08:57.594972 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.594939 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.595123 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.594992 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxnw\" (UniqueName: \"kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.595123 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.595030 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.595570 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.595546 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.597295 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.597276 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.647878 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.647851 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxnw\" (UniqueName: \"kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw\") pod \"maas-api-revoke-sub-c1e271a9-fpfwg\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:57.731725 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:57.731695 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:08:58.094713 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:58.094680 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg"] Nov 01 13:08:58.097689 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:08:58.097653 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164393ce_d1ef_4de3_ad6b_84e4ac874d52.slice/crio-8133e08eab35d96d726b4fdf382f86e3b865855179453734812ebd920b571ba4 WatchSource:0}: Error finding container 8133e08eab35d96d726b4fdf382f86e3b865855179453734812ebd920b571ba4: Status 404 returned error can't find the container with id 8133e08eab35d96d726b4fdf382f86e3b865855179453734812ebd920b571ba4 Nov 01 13:08:58.741502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:58.741462 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" event={"ID":"164393ce-d1ef-4de3-ad6b-84e4ac874d52","Type":"ContainerStarted","Data":"f653b79c23b9ed5d4285406ffc49e2c46fe467431b58e29fbed32cf44cebfc38"} Nov 01 13:08:58.741928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:58.741524 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" event={"ID":"164393ce-d1ef-4de3-ad6b-84e4ac874d52","Type":"ContainerStarted","Data":"8133e08eab35d96d726b4fdf382f86e3b865855179453734812ebd920b571ba4"} Nov 01 13:08:58.849830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:58.849782 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" podStartSLOduration=1.84976498 podStartE2EDuration="1.84976498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:08:58.848209224 +0000 UTC m=+2808.598152204" watchObservedRunningTime="2026-11-01 13:08:58.84976498 +0000 UTC m=+2808.599707987" Nov 01 13:08:59.747519 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:59.747488 4484 generic.go:356] "Generic (PLEG): container finished" podID="164393ce-d1ef-4de3-ad6b-84e4ac874d52" containerID="f653b79c23b9ed5d4285406ffc49e2c46fe467431b58e29fbed32cf44cebfc38" exitCode=0 Nov 01 13:08:59.747927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:08:59.747575 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" event={"ID":"164393ce-d1ef-4de3-ad6b-84e4ac874d52","Type":"ContainerDied","Data":"f653b79c23b9ed5d4285406ffc49e2c46fe467431b58e29fbed32cf44cebfc38"} Nov 01 13:09:00.250891 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:09:00.250856 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:09:00.883744 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.883718 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:09:00.926974 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.926943 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca\") pod \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " Nov 01 13:09:00.927120 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.926995 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca\") pod \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " Nov 01 13:09:00.927120 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.927113 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw\" (UniqueName: \"kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw\") pod \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\" (UID: \"164393ce-d1ef-4de3-ad6b-84e4ac874d52\") " Nov 01 13:09:00.927406 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.927384 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca" pod "164393ce-d1ef-4de3-ad6b-84e4ac874d52" (UID: "164393ce-d1ef-4de3-ad6b-84e4ac874d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:09:00.929060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.929040 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw" pod "164393ce-d1ef-4de3-ad6b-84e4ac874d52" (UID: "164393ce-d1ef-4de3-ad6b-84e4ac874d52"). InnerVolumeSpecName "kube-api-access-7gxnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:09:00.929125 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:00.929052 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca" pod "164393ce-d1ef-4de3-ad6b-84e4ac874d52" (UID: "164393ce-d1ef-4de3-ad6b-84e4ac874d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:09:01.028251 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:01.028174 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxnw\" (UniqueName: \"kubernetes.io/projected/164393ce-d1ef-4de3-ad6b-84e4ac874d52-kube-api-access-7gxnw\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:01.028251 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:01.028203 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:01.028251 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:01.028212 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164393ce-d1ef-4de3-ad6b-84e4ac874d52-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:01.758331 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:01.758288 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" Nov 01 13:09:01.758509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:01.758326 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg" event={"ID":"164393ce-d1ef-4de3-ad6b-84e4ac874d52","Type":"ContainerDied","Data":"8133e08eab35d96d726b4fdf382f86e3b865855179453734812ebd920b571ba4"} Nov 01 13:09:01.758509 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:01.758373 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8133e08eab35d96d726b4fdf382f86e3b865855179453734812ebd920b571ba4" Nov 01 13:09:09.052265 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:09.052225 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s"] Nov 01 13:09:09.098583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:09.098554 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdc138a-cdl5s"] Nov 01 13:09:10.319248 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:10.319214 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:09:10.366707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:10.366669 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9db29de-e28a-42b1-a896-9c82cd36a607" path="/var/lib/kubelet/pods/b9db29de-e28a-42b1-a896-9c82cd36a607/volumes" Nov 01 13:09:15.915226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:15.915171 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:09:21.239505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:21.239466 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:09:25.047479 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:25.047408 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt"] Nov 01 13:09:25.139649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:25.139619 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95943f4b-hgtgt"] Nov 01 13:09:26.368060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:26.368028 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98def0f2-4fae-4bd7-8a33-f288eef96e2d" path="/var/lib/kubelet/pods/98def0f2-4fae-4bd7-8a33-f288eef96e2d/volumes" Nov 01 13:09:28.388804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.388771 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx"] Nov 01 13:09:28.389195 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.389134 4484 cpu_manager.go:414] "containerMap: removing container" podUID="164393ce-d1ef-4de3-ad6b-84e4ac874d52" containerName="revoke-keys" Nov 01 13:09:28.389195 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.389145 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164393ce-d1ef-4de3-ad6b-84e4ac874d52" containerName="revoke-keys" Nov 01 13:09:28.389275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.389209 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="164393ce-d1ef-4de3-ad6b-84e4ac874d52" containerName="revoke-keys" Nov 01 13:09:28.392622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.392602 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.408097 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.408072 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.408205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.408127 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.408205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.408188 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2c4\" (UniqueName: \"kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.445985 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.445954 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx"] Nov 01 13:09:28.490131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.490105 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:09:28.490131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.490107 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:09:28.490388 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.490373 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:09:28.490475 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.490448 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:09:28.509043 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.509018 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.509137 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.509057 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2c4\" (UniqueName: \"kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.509190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.509155 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.509894 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.509877 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.511374 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.511358 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.562849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.562824 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2c4\" (UniqueName: \"kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4\") pod \"maas-api-revoke-sub-2e1f5b5f-sshcx\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:28.704851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:28.704772 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:29.072366 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:29.072335 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx"] Nov 01 13:09:29.075546 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:09:29.075513 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d25f15_805f_4684_b9f0_c6680a97474e.slice/crio-60b52c30d2465d857d00c6c26fb0774d25d774ced0e5f5259cf70646b776272d WatchSource:0}: Error finding container 60b52c30d2465d857d00c6c26fb0774d25d774ced0e5f5259cf70646b776272d: Status 404 returned error can't find the container with id 60b52c30d2465d857d00c6c26fb0774d25d774ced0e5f5259cf70646b776272d Nov 01 13:09:29.883467 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:29.883431 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" event={"ID":"b8d25f15-805f-4684-b9f0-c6680a97474e","Type":"ContainerStarted","Data":"27b65ff6ef83b5e3f3607de9554684550a1ec09eff45cffb1d8cae0bcc76ec20"} Nov 01 13:09:29.883878 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:29.883494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" event={"ID":"b8d25f15-805f-4684-b9f0-c6680a97474e","Type":"ContainerStarted","Data":"60b52c30d2465d857d00c6c26fb0774d25d774ced0e5f5259cf70646b776272d"} Nov 01 13:09:29.990717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:29.990669 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" podStartSLOduration=1.990654294 podStartE2EDuration="1.990654294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:09:29.988995408 +0000 UTC m=+2839.738938386" watchObservedRunningTime="2026-11-01 13:09:29.990654294 +0000 UTC m=+2839.740597272" Nov 01 13:09:30.250205 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:09:30.250159 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:09:30.889832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:30.889800 4484 generic.go:356] "Generic (PLEG): container finished" podID="b8d25f15-805f-4684-b9f0-c6680a97474e" containerID="27b65ff6ef83b5e3f3607de9554684550a1ec09eff45cffb1d8cae0bcc76ec20" exitCode=0 Nov 01 13:09:30.890232 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:30.889884 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" event={"ID":"b8d25f15-805f-4684-b9f0-c6680a97474e","Type":"ContainerDied","Data":"27b65ff6ef83b5e3f3607de9554684550a1ec09eff45cffb1d8cae0bcc76ec20"} Nov 01 13:09:32.024665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.024638 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:32.144776 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.144714 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4\" (UniqueName: \"kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4\") pod \"b8d25f15-805f-4684-b9f0-c6680a97474e\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " Nov 01 13:09:32.144776 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.144752 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca\") pod \"b8d25f15-805f-4684-b9f0-c6680a97474e\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " Nov 01 13:09:32.145012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.144809 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca\") pod \"b8d25f15-805f-4684-b9f0-c6680a97474e\" (UID: \"b8d25f15-805f-4684-b9f0-c6680a97474e\") " Nov 01 13:09:32.145232 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.145210 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca" pod "b8d25f15-805f-4684-b9f0-c6680a97474e" (UID: "b8d25f15-805f-4684-b9f0-c6680a97474e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:09:32.146854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.146827 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4" pod "b8d25f15-805f-4684-b9f0-c6680a97474e" (UID: "b8d25f15-805f-4684-b9f0-c6680a97474e"). InnerVolumeSpecName "kube-api-access-8d2c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:09:32.146854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.146845 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca" pod "b8d25f15-805f-4684-b9f0-c6680a97474e" (UID: "b8d25f15-805f-4684-b9f0-c6680a97474e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:09:32.245426 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.245396 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:32.245426 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.245421 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2c4\" (UniqueName: \"kubernetes.io/projected/b8d25f15-805f-4684-b9f0-c6680a97474e-kube-api-access-8d2c4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:32.245570 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.245436 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d25f15-805f-4684-b9f0-c6680a97474e-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:32.899396 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.899357 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" Nov 01 13:09:32.899396 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.899389 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx" event={"ID":"b8d25f15-805f-4684-b9f0-c6680a97474e","Type":"ContainerDied","Data":"60b52c30d2465d857d00c6c26fb0774d25d774ced0e5f5259cf70646b776272d"} Nov 01 13:09:32.899613 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:32.899438 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b52c30d2465d857d00c6c26fb0774d25d774ced0e5f5259cf70646b776272d" Nov 01 13:09:40.716150 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.716114 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc"] Nov 01 13:09:40.716792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.716531 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b8d25f15-805f-4684-b9f0-c6680a97474e" containerName="revoke-keys" Nov 01 13:09:40.716792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.716545 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d25f15-805f-4684-b9f0-c6680a97474e" containerName="revoke-keys" Nov 01 13:09:40.716792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.716613 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d25f15-805f-4684-b9f0-c6680a97474e" containerName="revoke-keys" Nov 01 13:09:40.719877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.719855 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.772486 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.772451 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc"] Nov 01 13:09:40.816064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.816035 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:09:40.816326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.816294 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:09:40.816391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.816335 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:09:40.816391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.816348 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:09:40.823695 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.823671 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.823781 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.823705 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5mx\" (UniqueName: \"kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.823820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.823803 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.924585 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.924551 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.924585 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.924590 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5mx\" (UniqueName: \"kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.924768 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.924712 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.925225 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.925199 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.927008 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.926979 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:40.976981 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:40.976956 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5mx\" (UniqueName: \"kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx\") pod \"maas-api-revoke-sub-ec843018-22dtc\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:41.033231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:41.033208 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:41.393170 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:41.393092 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc"] Nov 01 13:09:41.396093 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:09:41.396064 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a098303_701b_48bc_b79e_784a3886d80c.slice/crio-1eb86854bed757ad2ce760a2bacb2ce217cd4bba23db1f13690a5259a3f2baa5 WatchSource:0}: Error finding container 1eb86854bed757ad2ce760a2bacb2ce217cd4bba23db1f13690a5259a3f2baa5: Status 404 returned error can't find the container with id 1eb86854bed757ad2ce760a2bacb2ce217cd4bba23db1f13690a5259a3f2baa5 Nov 01 13:09:41.944411 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:41.944330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" event={"ID":"1a098303-701b-48bc-b79e-784a3886d80c","Type":"ContainerStarted","Data":"01afc055cfdf1a4a0ed9a1585ae64be4fd31041775da17661f15ae4840d8ef82"} Nov 01 13:09:41.944411 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:41.944394 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" event={"ID":"1a098303-701b-48bc-b79e-784a3886d80c","Type":"ContainerStarted","Data":"1eb86854bed757ad2ce760a2bacb2ce217cd4bba23db1f13690a5259a3f2baa5"} Nov 01 13:09:42.052404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:42.052349 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" podStartSLOduration=2.052335407 podStartE2EDuration="2.052335407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:09:42.049768294 +0000 UTC m=+2851.799711271" watchObservedRunningTime="2026-11-01 13:09:42.052335407 +0000 UTC m=+2851.802278382" Nov 01 13:09:42.949988 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:42.949954 4484 generic.go:356] "Generic (PLEG): container finished" podID="1a098303-701b-48bc-b79e-784a3886d80c" containerID="01afc055cfdf1a4a0ed9a1585ae64be4fd31041775da17661f15ae4840d8ef82" exitCode=0 Nov 01 13:09:42.950384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:42.950020 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" event={"ID":"1a098303-701b-48bc-b79e-784a3886d80c","Type":"ContainerDied","Data":"01afc055cfdf1a4a0ed9a1585ae64be4fd31041775da17661f15ae4840d8ef82"} Nov 01 13:09:44.083188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.083165 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:44.154652 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.154624 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca\") pod \"1a098303-701b-48bc-b79e-784a3886d80c\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " Nov 01 13:09:44.154766 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.154749 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca\") pod \"1a098303-701b-48bc-b79e-784a3886d80c\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " Nov 01 13:09:44.154816 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.154805 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx\" (UniqueName: \"kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx\") pod \"1a098303-701b-48bc-b79e-784a3886d80c\" (UID: \"1a098303-701b-48bc-b79e-784a3886d80c\") " Nov 01 13:09:44.155076 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.155053 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca" pod "1a098303-701b-48bc-b79e-784a3886d80c" (UID: "1a098303-701b-48bc-b79e-784a3886d80c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:09:44.156757 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.156736 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx" pod "1a098303-701b-48bc-b79e-784a3886d80c" (UID: "1a098303-701b-48bc-b79e-784a3886d80c"). InnerVolumeSpecName "kube-api-access-rk5mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:09:44.156826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.156812 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca" pod "1a098303-701b-48bc-b79e-784a3886d80c" (UID: "1a098303-701b-48bc-b79e-784a3886d80c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:09:44.255881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.255855 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:44.255881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.255879 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk5mx\" (UniqueName: \"kubernetes.io/projected/1a098303-701b-48bc-b79e-784a3886d80c-kube-api-access-rk5mx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:44.256035 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.255889 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a098303-701b-48bc-b79e-784a3886d80c-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:09:44.914852 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.914819 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:09:44.960503 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.960472 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" Nov 01 13:09:44.960676 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.960519 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc" event={"ID":"1a098303-701b-48bc-b79e-784a3886d80c","Type":"ContainerDied","Data":"1eb86854bed757ad2ce760a2bacb2ce217cd4bba23db1f13690a5259a3f2baa5"} Nov 01 13:09:44.960676 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:44.960568 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb86854bed757ad2ce760a2bacb2ce217cd4bba23db1f13690a5259a3f2baa5" Nov 01 13:09:57.811192 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:09:57.811157 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:10:00.315167 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:10:00.315129 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:10:06.917003 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:06.916973 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:10:07.034314 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:07.034276 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:10:12.113316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:12.113269 4484 scope.go:122] "RemoveContainer" containerID="901482421abbfac7f4619cf8307c6b7d0f550dd82d40d5f06da320db00c54b4f" Nov 01 13:10:15.288911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.288879 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j"] Nov 01 13:10:15.289449 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.289428 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1a098303-701b-48bc-b79e-784a3886d80c" containerName="revoke-keys" Nov 01 13:10:15.289535 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.289452 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a098303-701b-48bc-b79e-784a3886d80c" containerName="revoke-keys" Nov 01 13:10:15.289593 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.289571 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a098303-701b-48bc-b79e-784a3886d80c" containerName="revoke-keys" Nov 01 13:10:15.294149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.294128 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.337931 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.337904 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcb98\" (UniqueName: \"kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.338045 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.337946 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.338088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.338066 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.345189 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.345163 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j"] Nov 01 13:10:15.388835 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.388800 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:10:15.388963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.388842 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:10:15.388963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.388873 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:10:15.388963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.388923 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:10:15.439308 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.439280 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcb98\" (UniqueName: \"kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.439429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.439348 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.439429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.439421 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.439987 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.439966 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.441686 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.441667 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.491771 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.491749 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcb98\" (UniqueName: \"kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98\") pod \"maas-api-revoke-sub-046e1343-86g4j\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.606736 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.606685 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:15.968345 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:15.968253 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j"] Nov 01 13:10:15.972349 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:10:15.972266 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5720cd_0966_428b_b26b_666376855307.slice/crio-83e4bfaa962b1c3ac99fd8e475d0bb76574ef7baf2dc4334ee6ec52c541d10f7 WatchSource:0}: Error finding container 83e4bfaa962b1c3ac99fd8e475d0bb76574ef7baf2dc4334ee6ec52c541d10f7: Status 404 returned error can't find the container with id 83e4bfaa962b1c3ac99fd8e475d0bb76574ef7baf2dc4334ee6ec52c541d10f7 Nov 01 13:10:16.124358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:16.124330 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" event={"ID":"9e5720cd-0966-428b-b26b-666376855307","Type":"ContainerStarted","Data":"83e4bfaa962b1c3ac99fd8e475d0bb76574ef7baf2dc4334ee6ec52c541d10f7"} Nov 01 13:10:16.237151 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:16.237120 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:10:17.130503 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:17.130416 4484 generic.go:356] "Generic (PLEG): container finished" podID="9e5720cd-0966-428b-b26b-666376855307" containerID="10c5df29b75c074a3600fef575c07e7adf97b201daa2f6a45385716b03f3b23b" exitCode=0 Nov 01 13:10:17.130856 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:17.130504 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" event={"ID":"9e5720cd-0966-428b-b26b-666376855307","Type":"ContainerDied","Data":"10c5df29b75c074a3600fef575c07e7adf97b201daa2f6a45385716b03f3b23b"} Nov 01 13:10:18.263845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.263823 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:18.367664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.367643 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca\") pod \"9e5720cd-0966-428b-b26b-666376855307\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " Nov 01 13:10:18.367801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.367674 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca\") pod \"9e5720cd-0966-428b-b26b-666376855307\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " Nov 01 13:10:18.367879 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.367821 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98\" (UniqueName: \"kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98\") pod \"9e5720cd-0966-428b-b26b-666376855307\" (UID: \"9e5720cd-0966-428b-b26b-666376855307\") " Nov 01 13:10:18.368047 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.368022 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca" pod "9e5720cd-0966-428b-b26b-666376855307" (UID: "9e5720cd-0966-428b-b26b-666376855307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:10:18.369790 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.369766 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98" pod "9e5720cd-0966-428b-b26b-666376855307" (UID: "9e5720cd-0966-428b-b26b-666376855307"). InnerVolumeSpecName "kube-api-access-dcb98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:10:18.369993 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.369973 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca" pod "9e5720cd-0966-428b-b26b-666376855307" (UID: "9e5720cd-0966-428b-b26b-666376855307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:10:18.468922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.468854 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcb98\" (UniqueName: \"kubernetes.io/projected/9e5720cd-0966-428b-b26b-666376855307-kube-api-access-dcb98\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:10:18.468922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.468885 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e5720cd-0966-428b-b26b-666376855307-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:10:18.468922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:18.468903 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e5720cd-0966-428b-b26b-666376855307-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:10:19.145834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:19.145799 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" event={"ID":"9e5720cd-0966-428b-b26b-666376855307","Type":"ContainerDied","Data":"83e4bfaa962b1c3ac99fd8e475d0bb76574ef7baf2dc4334ee6ec52c541d10f7"} Nov 01 13:10:19.146012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:19.145874 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e4bfaa962b1c3ac99fd8e475d0bb76574ef7baf2dc4334ee6ec52c541d10f7" Nov 01 13:10:19.146012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:19.145809 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j" Nov 01 13:10:25.659854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.659801 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:10:25.907970 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.907924 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh"] Nov 01 13:10:25.908367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.908351 4484 cpu_manager.go:414] "containerMap: removing container" podUID="9e5720cd-0966-428b-b26b-666376855307" containerName="revoke-keys" Nov 01 13:10:25.908367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.908366 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5720cd-0966-428b-b26b-666376855307" containerName="revoke-keys" Nov 01 13:10:25.908540 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.908439 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5720cd-0966-428b-b26b-666376855307" containerName="revoke-keys" Nov 01 13:10:25.918090 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.918015 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:25.965954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:25.965920 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh"] Nov 01 13:10:26.009799 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.009766 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:10:26.009963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.009877 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:10:26.009963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.009888 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:10:26.012785 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.012773 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:10:26.033241 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.033214 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.033392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.033267 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvwr\" (UniqueName: \"kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.033392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.033385 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.134260 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.134230 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.134408 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.134287 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvwr\" (UniqueName: \"kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.134457 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.134420 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.134976 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.134955 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.136785 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.136764 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.188797 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.188737 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvwr\" (UniqueName: \"kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr\") pod \"maas-api-revoke-sub-d3fba056-gbzxh\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.236792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.236763 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:26.601457 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:26.601429 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh"] Nov 01 13:10:26.604011 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:10:26.603982 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3366700_e1fa_4720_ac8b_126f9d9efc85.slice/crio-5876d08c58e2564bc82ae4a2ce91b914dbe16491deaca39554f9b052cc5e2dc3 WatchSource:0}: Error finding container 5876d08c58e2564bc82ae4a2ce91b914dbe16491deaca39554f9b052cc5e2dc3: Status 404 returned error can't find the container with id 5876d08c58e2564bc82ae4a2ce91b914dbe16491deaca39554f9b052cc5e2dc3 Nov 01 13:10:27.184041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:27.183954 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" event={"ID":"b3366700-e1fa-4720-ac8b-126f9d9efc85","Type":"ContainerStarted","Data":"91ed73c3ff41b1ca79cda17af3fa421cc251cdb163030170bb35cfc2b5ce35d8"} Nov 01 13:10:27.184041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:27.184013 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" event={"ID":"b3366700-e1fa-4720-ac8b-126f9d9efc85","Type":"ContainerStarted","Data":"5876d08c58e2564bc82ae4a2ce91b914dbe16491deaca39554f9b052cc5e2dc3"} Nov 01 13:10:27.291259 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:27.291208 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" podStartSLOduration=2.29119596 podStartE2EDuration="2.29119596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:10:27.290617901 +0000 UTC m=+2897.040560878" watchObservedRunningTime="2026-11-01 13:10:27.29119596 +0000 UTC m=+2897.041138938" Nov 01 13:10:28.135161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:28.135124 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:10:28.190453 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:28.190416 4484 generic.go:356] "Generic (PLEG): container finished" podID="b3366700-e1fa-4720-ac8b-126f9d9efc85" containerID="91ed73c3ff41b1ca79cda17af3fa421cc251cdb163030170bb35cfc2b5ce35d8" exitCode=0 Nov 01 13:10:28.190921 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:28.190504 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" event={"ID":"b3366700-e1fa-4720-ac8b-126f9d9efc85","Type":"ContainerDied","Data":"91ed73c3ff41b1ca79cda17af3fa421cc251cdb163030170bb35cfc2b5ce35d8"} Nov 01 13:10:29.352534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.352494 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:29.489376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.477281 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr\" (UniqueName: \"kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr\") pod \"b3366700-e1fa-4720-ac8b-126f9d9efc85\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " Nov 01 13:10:29.489376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.477361 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca\") pod \"b3366700-e1fa-4720-ac8b-126f9d9efc85\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " Nov 01 13:10:29.489376 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.477392 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca\") pod \"b3366700-e1fa-4720-ac8b-126f9d9efc85\" (UID: \"b3366700-e1fa-4720-ac8b-126f9d9efc85\") " Nov 01 13:10:29.498361 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.497728 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr" pod "b3366700-e1fa-4720-ac8b-126f9d9efc85" (UID: "b3366700-e1fa-4720-ac8b-126f9d9efc85"). InnerVolumeSpecName "kube-api-access-pqvwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:10:29.506454 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.498645 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca" pod "b3366700-e1fa-4720-ac8b-126f9d9efc85" (UID: "b3366700-e1fa-4720-ac8b-126f9d9efc85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:10:29.519369 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.506904 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca" pod "b3366700-e1fa-4720-ac8b-126f9d9efc85" (UID: "b3366700-e1fa-4720-ac8b-126f9d9efc85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:10:29.584323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.580872 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:10:29.584323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.580935 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3366700-e1fa-4720-ac8b-126f9d9efc85-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:10:29.584323 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:29.580954 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqvwr\" (UniqueName: \"kubernetes.io/projected/b3366700-e1fa-4720-ac8b-126f9d9efc85-kube-api-access-pqvwr\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:10:30.213851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:30.213752 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" event={"ID":"b3366700-e1fa-4720-ac8b-126f9d9efc85","Type":"ContainerDied","Data":"5876d08c58e2564bc82ae4a2ce91b914dbe16491deaca39554f9b052cc5e2dc3"} Nov 01 13:10:30.213851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:30.213822 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5876d08c58e2564bc82ae4a2ce91b914dbe16491deaca39554f9b052cc5e2dc3" Nov 01 13:10:30.214211 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:30.214183 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh" Nov 01 13:10:30.315333 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:10:30.312452 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:10:38.649608 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:38.649573 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:10:49.311739 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:49.311705 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:10:49.431792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:49.431758 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:10:49.644218 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:10:49.644142 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:11:00.250942 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:11:00.250861 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:11:00.455113 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.455083 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs"] Nov 01 13:11:00.455474 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.455460 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b3366700-e1fa-4720-ac8b-126f9d9efc85" containerName="revoke-keys" Nov 01 13:11:00.455529 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.455475 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3366700-e1fa-4720-ac8b-126f9d9efc85" containerName="revoke-keys" Nov 01 13:11:00.455574 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.455541 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3366700-e1fa-4720-ac8b-126f9d9efc85" containerName="revoke-keys" Nov 01 13:11:00.465260 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.465232 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.512641 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.512569 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs"] Nov 01 13:11:00.525002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.524973 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxhz\" (UniqueName: \"kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.525140 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.525016 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.525209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.525182 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.555959 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.555931 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:11:00.556075 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.555961 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:11:00.556075 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.555980 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:11:00.559289 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.559274 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:11:00.626476 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.626450 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxhz\" (UniqueName: \"kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.626586 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.626488 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.626586 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.626539 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.627123 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.627106 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.628709 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.628692 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.679006 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.678979 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxhz\" (UniqueName: \"kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz\") pod \"maas-api-revoke-sub-ae594979-jknxs\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:00.781018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:00.780958 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:01.144573 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:01.144472 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs"] Nov 01 13:11:01.147232 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:11:01.147197 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60da4116_4958_49e6_a0d9_bc5a52fcf17e.slice/crio-f78b581cafbde5a279a6462901c9cdfd82823516ab6a385772f1aca04d460d68 WatchSource:0}: Error finding container f78b581cafbde5a279a6462901c9cdfd82823516ab6a385772f1aca04d460d68: Status 404 returned error can't find the container with id f78b581cafbde5a279a6462901c9cdfd82823516ab6a385772f1aca04d460d68 Nov 01 13:11:01.360781 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:01.360747 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" event={"ID":"60da4116-4958-49e6-a0d9-bc5a52fcf17e","Type":"ContainerStarted","Data":"f78b581cafbde5a279a6462901c9cdfd82823516ab6a385772f1aca04d460d68"} Nov 01 13:11:02.366879 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:02.366846 4484 generic.go:356] "Generic (PLEG): container finished" podID="60da4116-4958-49e6-a0d9-bc5a52fcf17e" containerID="175cb42b41b1ef3bc7f060c9cb8710c78a07868f88c44a650d6d7144747513d5" exitCode=0 Nov 01 13:11:02.368384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:02.368355 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" event={"ID":"60da4116-4958-49e6-a0d9-bc5a52fcf17e","Type":"ContainerDied","Data":"175cb42b41b1ef3bc7f060c9cb8710c78a07868f88c44a650d6d7144747513d5"} Nov 01 13:11:03.500085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.500061 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:03.551855 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.551828 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca\") pod \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " Nov 01 13:11:03.551975 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.551919 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca\") pod \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " Nov 01 13:11:03.552050 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.551982 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz\" (UniqueName: \"kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz\") pod \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\" (UID: \"60da4116-4958-49e6-a0d9-bc5a52fcf17e\") " Nov 01 13:11:03.552269 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.552247 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca" pod "60da4116-4958-49e6-a0d9-bc5a52fcf17e" (UID: "60da4116-4958-49e6-a0d9-bc5a52fcf17e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:11:03.553850 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.553823 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca" pod "60da4116-4958-49e6-a0d9-bc5a52fcf17e" (UID: "60da4116-4958-49e6-a0d9-bc5a52fcf17e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:11:03.553951 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.553875 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz" pod "60da4116-4958-49e6-a0d9-bc5a52fcf17e" (UID: "60da4116-4958-49e6-a0d9-bc5a52fcf17e"). InnerVolumeSpecName "kube-api-access-2bxhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:11:03.653575 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.653522 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxhz\" (UniqueName: \"kubernetes.io/projected/60da4116-4958-49e6-a0d9-bc5a52fcf17e-kube-api-access-2bxhz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:11:03.653575 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.653544 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:11:03.653575 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:03.653555 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da4116-4958-49e6-a0d9-bc5a52fcf17e-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:11:04.376755 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:04.376730 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" Nov 01 13:11:04.376908 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:04.376724 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs" event={"ID":"60da4116-4958-49e6-a0d9-bc5a52fcf17e","Type":"ContainerDied","Data":"f78b581cafbde5a279a6462901c9cdfd82823516ab6a385772f1aca04d460d68"} Nov 01 13:11:04.376908 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:04.376849 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78b581cafbde5a279a6462901c9cdfd82823516ab6a385772f1aca04d460d68" Nov 01 13:11:10.245553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:10.245518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:11:12.134337 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:12.134283 4484 scope.go:122] "RemoveContainer" containerID="edeafffd3507a81a0f5c9624d65aae26844d12a68713e1189c5b13efe153c4ad" Nov 01 13:11:13.322359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:13.322325 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:11:16.572862 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:16.572827 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:11:23.031446 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:23.031411 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:11:28.346486 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.346452 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4"] Nov 01 13:11:28.346961 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.346842 4484 cpu_manager.go:414] "containerMap: removing container" podUID="60da4116-4958-49e6-a0d9-bc5a52fcf17e" containerName="revoke-keys" Nov 01 13:11:28.346961 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.346852 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60da4116-4958-49e6-a0d9-bc5a52fcf17e" containerName="revoke-keys" Nov 01 13:11:28.346961 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.346921 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="60da4116-4958-49e6-a0d9-bc5a52fcf17e" containerName="revoke-keys" Nov 01 13:11:28.351513 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.351484 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.401197 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.401162 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.401413 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.401363 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.401413 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.401407 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhld9\" (UniqueName: \"kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.446453 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.446416 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:11:28.446453 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.446416 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:11:28.446453 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.446417 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:11:28.446671 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.446442 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:11:28.448420 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.448400 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4"] Nov 01 13:11:28.501871 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.501841 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.502039 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.501932 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.502039 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.501949 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhld9\" (UniqueName: \"kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.502500 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.502481 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.504192 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.504172 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.554532 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.554504 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhld9\" (UniqueName: \"kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9\") pod \"maas-api-revoke-sub-a4aef7d7-kt2p4\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:28.663462 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:28.663382 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:29.033389 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:29.033360 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4"] Nov 01 13:11:29.035632 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:11:29.035602 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba370f5e_c6ce_4916_90a3_b5e8ec0bff49.slice/crio-284fb99819ae210fcba1153d36f6ae63795011c9ecb106e0accb7c0b8a1328d2 WatchSource:0}: Error finding container 284fb99819ae210fcba1153d36f6ae63795011c9ecb106e0accb7c0b8a1328d2: Status 404 returned error can't find the container with id 284fb99819ae210fcba1153d36f6ae63795011c9ecb106e0accb7c0b8a1328d2 Nov 01 13:11:29.508012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:29.507976 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" event={"ID":"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49","Type":"ContainerStarted","Data":"cf93f525e989a5df161b2d3f1f80b73224cca72fe5f56196acfefd1f10a8399d"} Nov 01 13:11:29.508515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:29.508038 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" event={"ID":"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49","Type":"ContainerStarted","Data":"284fb99819ae210fcba1153d36f6ae63795011c9ecb106e0accb7c0b8a1328d2"} Nov 01 13:11:29.614134 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:29.614079 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" podStartSLOduration=1.614066257 podStartE2EDuration="1.614066257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:11:29.613406528 +0000 UTC m=+2959.363349514" watchObservedRunningTime="2026-11-01 13:11:29.614066257 +0000 UTC m=+2959.364009235" Nov 01 13:11:30.250450 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:11:30.250415 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:11:30.514514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:30.514430 4484 generic.go:356] "Generic (PLEG): container finished" podID="ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" containerID="cf93f525e989a5df161b2d3f1f80b73224cca72fe5f56196acfefd1f10a8399d" exitCode=0 Nov 01 13:11:30.514514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:30.514482 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" event={"ID":"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49","Type":"ContainerDied","Data":"cf93f525e989a5df161b2d3f1f80b73224cca72fe5f56196acfefd1f10a8399d"} Nov 01 13:11:31.653152 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.653124 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:31.730947 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.730915 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca\") pod \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " Nov 01 13:11:31.731121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.731007 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9\" (UniqueName: \"kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9\") pod \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " Nov 01 13:11:31.731121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.731079 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca\") pod \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\" (UID: \"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49\") " Nov 01 13:11:31.731401 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.731364 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca" pod "ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" (UID: "ba370f5e-c6ce-4916-90a3-b5e8ec0bff49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:11:31.733770 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.733746 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca" pod "ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" (UID: "ba370f5e-c6ce-4916-90a3-b5e8ec0bff49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:11:31.733866 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.733772 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9" pod "ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" (UID: "ba370f5e-c6ce-4916-90a3-b5e8ec0bff49"). InnerVolumeSpecName "kube-api-access-mhld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:11:31.832862 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.832781 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhld9\" (UniqueName: \"kubernetes.io/projected/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-kube-api-access-mhld9\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:11:31.832862 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.832811 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:11:31.832862 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:31.832821 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:11:32.525158 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:32.525113 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" event={"ID":"ba370f5e-c6ce-4916-90a3-b5e8ec0bff49","Type":"ContainerDied","Data":"284fb99819ae210fcba1153d36f6ae63795011c9ecb106e0accb7c0b8a1328d2"} Nov 01 13:11:32.525158 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:32.525136 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4" Nov 01 13:11:32.525402 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:32.525199 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284fb99819ae210fcba1153d36f6ae63795011c9ecb106e0accb7c0b8a1328d2" Nov 01 13:11:41.223848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:41.223814 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:11:53.726368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:53.726323 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:11:58.761238 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.761199 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8"] Nov 01 13:11:58.761786 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.761764 4484 cpu_manager.go:414] "containerMap: removing container" podUID="ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" containerName="revoke-keys" Nov 01 13:11:58.761873 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.761788 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" containerName="revoke-keys" Nov 01 13:11:58.761932 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.761913 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" containerName="revoke-keys" Nov 01 13:11:58.765132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.765111 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.799743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.799713 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.799878 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.799750 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbbm\" (UniqueName: \"kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.799878 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.799774 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.861495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.861434 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:11:58.861495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.861469 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:11:58.861704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.861658 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:11:58.861704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.861687 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:11:58.863738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.863717 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8"] Nov 01 13:11:58.901102 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.901072 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.901249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.901105 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbbm\" (UniqueName: \"kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.901249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.901132 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.901841 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.901816 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.903445 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.903427 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:58.953773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:58.953747 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbbm\" (UniqueName: \"kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm\") pod \"maas-api-revoke-sub-79805ca9-5bnb8\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:59.053241 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:59.053138 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml"] Nov 01 13:11:59.055800 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:59.055736 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c804ef7c-hnhml"] Nov 01 13:11:59.080003 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:59.079976 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:11:59.445146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:59.445120 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8"] Nov 01 13:11:59.447872 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:11:59.447814 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb3c75e_2383_4536_aa9a_4a0d42c143ec.slice/crio-6fdff3ab5316bfc8fa1784b09871e6aef1c64dbdd101cbbb5c987274a9e67c6f WatchSource:0}: Error finding container 6fdff3ab5316bfc8fa1784b09871e6aef1c64dbdd101cbbb5c987274a9e67c6f: Status 404 returned error can't find the container with id 6fdff3ab5316bfc8fa1784b09871e6aef1c64dbdd101cbbb5c987274a9e67c6f Nov 01 13:11:59.661226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:11:59.661194 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" event={"ID":"4eb3c75e-2383-4536-aa9a-4a0d42c143ec","Type":"ContainerStarted","Data":"6fdff3ab5316bfc8fa1784b09871e6aef1c64dbdd101cbbb5c987274a9e67c6f"} Nov 01 13:12:00.252780 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:12:00.252740 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:12:00.367579 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:00.367541 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3896110a-ad91-4113-a6e4-f3d620c0ac22" path="/var/lib/kubelet/pods/3896110a-ad91-4113-a6e4-f3d620c0ac22/volumes" Nov 01 13:12:00.668416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:00.668330 4484 generic.go:356] "Generic (PLEG): container finished" podID="4eb3c75e-2383-4536-aa9a-4a0d42c143ec" containerID="04bdb71a03db86f03df268a8e19cd0bf4475fa890c2493370b6cc62e78b5acac" exitCode=0 Nov 01 13:12:00.668613 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:00.668415 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" event={"ID":"4eb3c75e-2383-4536-aa9a-4a0d42c143ec","Type":"ContainerDied","Data":"04bdb71a03db86f03df268a8e19cd0bf4475fa890c2493370b6cc62e78b5acac"} Nov 01 13:12:01.810949 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.810925 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:12:01.928157 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.928079 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca\") pod \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " Nov 01 13:12:01.928157 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.928145 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca\") pod \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " Nov 01 13:12:01.928404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.928261 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm\" (UniqueName: \"kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm\") pod \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\" (UID: \"4eb3c75e-2383-4536-aa9a-4a0d42c143ec\") " Nov 01 13:12:01.928554 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.928529 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca" pod "4eb3c75e-2383-4536-aa9a-4a0d42c143ec" (UID: "4eb3c75e-2383-4536-aa9a-4a0d42c143ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:12:01.928679 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.928661 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:01.930314 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.930276 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm" pod "4eb3c75e-2383-4536-aa9a-4a0d42c143ec" (UID: "4eb3c75e-2383-4536-aa9a-4a0d42c143ec"). InnerVolumeSpecName "kube-api-access-csbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:12:01.930702 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:01.930682 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca" pod "4eb3c75e-2383-4536-aa9a-4a0d42c143ec" (UID: "4eb3c75e-2383-4536-aa9a-4a0d42c143ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:12:02.029803 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:02.029772 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:02.029803 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:02.029800 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbbm\" (UniqueName: \"kubernetes.io/projected/4eb3c75e-2383-4536-aa9a-4a0d42c143ec-kube-api-access-csbbm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:02.679956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:02.679874 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" Nov 01 13:12:02.679956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:02.679879 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8" event={"ID":"4eb3c75e-2383-4536-aa9a-4a0d42c143ec","Type":"ContainerDied","Data":"6fdff3ab5316bfc8fa1784b09871e6aef1c64dbdd101cbbb5c987274a9e67c6f"} Nov 01 13:12:02.679956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:02.679922 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdff3ab5316bfc8fa1784b09871e6aef1c64dbdd101cbbb5c987274a9e67c6f" Nov 01 13:12:12.155745 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:12.155714 4484 scope.go:122] "RemoveContainer" containerID="256914326ede6e818ce7da8d0373a997f8035f30896e6a46ba387fd5fd32cbb5" Nov 01 13:12:13.441880 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:13.441846 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:12:16.955040 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:16.955003 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:12:22.472066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.472033 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg"] Nov 01 13:12:22.472502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.472411 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4eb3c75e-2383-4536-aa9a-4a0d42c143ec" containerName="revoke-keys" Nov 01 13:12:22.472502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.472422 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb3c75e-2383-4536-aa9a-4a0d42c143ec" containerName="revoke-keys" Nov 01 13:12:22.472502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.472490 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb3c75e-2383-4536-aa9a-4a0d42c143ec" containerName="revoke-keys" Nov 01 13:12:22.475596 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.475567 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.573743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.573712 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:12:22.573975 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.573745 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:12:22.573975 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.573803 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:12:22.574180 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.574006 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:12:22.577612 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.577587 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg"] Nov 01 13:12:22.657214 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.657187 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.657393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.657229 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.657393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.657254 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2crk\" (UniqueName: \"kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.757885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.757860 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.758064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.757894 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2crk\" (UniqueName: \"kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.758064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.757973 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.758577 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.758552 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.760106 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.760085 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:22.810756 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:22.810732 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2crk\" (UniqueName: \"kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk\") pod \"maas-api-revoke-sub-0b7e8edb-v7cdg\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:23.088512 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:23.088416 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:23.452690 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:23.452667 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg"] Nov 01 13:12:23.454531 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:12:23.454495 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd72f84d_a962_40d7_805b_ad3344a7e13c.slice/crio-ae0d652fbe19dd72dda1a2730ca68c629ce54fb5657cfa08d196b8468d158bc5 WatchSource:0}: Error finding container ae0d652fbe19dd72dda1a2730ca68c629ce54fb5657cfa08d196b8468d158bc5: Status 404 returned error can't find the container with id ae0d652fbe19dd72dda1a2730ca68c629ce54fb5657cfa08d196b8468d158bc5 Nov 01 13:12:23.782816 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:23.782785 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" event={"ID":"bd72f84d-a962-40d7-805b-ad3344a7e13c","Type":"ContainerStarted","Data":"eaeedcd348c0cf34319e18972811b0311e15b2c4999b1f1dee7c240da693d07b"} Nov 01 13:12:23.783121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:23.782839 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" event={"ID":"bd72f84d-a962-40d7-805b-ad3344a7e13c","Type":"ContainerStarted","Data":"ae0d652fbe19dd72dda1a2730ca68c629ce54fb5657cfa08d196b8468d158bc5"} Nov 01 13:12:23.889257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:23.889209 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" podStartSLOduration=1.889193005 podStartE2EDuration="1.889193005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:12:23.887893224 +0000 UTC m=+3013.637836202" watchObservedRunningTime="2026-11-01 13:12:23.889193005 +0000 UTC m=+3013.639135984" Nov 01 13:12:24.789285 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:24.789248 4484 generic.go:356] "Generic (PLEG): container finished" podID="bd72f84d-a962-40d7-805b-ad3344a7e13c" containerID="eaeedcd348c0cf34319e18972811b0311e15b2c4999b1f1dee7c240da693d07b" exitCode=0 Nov 01 13:12:24.789709 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:24.789336 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" event={"ID":"bd72f84d-a962-40d7-805b-ad3344a7e13c","Type":"ContainerDied","Data":"eaeedcd348c0cf34319e18972811b0311e15b2c4999b1f1dee7c240da693d07b"} Nov 01 13:12:25.915847 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:25.915819 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:26.088861 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.088782 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk\" (UniqueName: \"kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk\") pod \"bd72f84d-a962-40d7-805b-ad3344a7e13c\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " Nov 01 13:12:26.089011 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.088918 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca\") pod \"bd72f84d-a962-40d7-805b-ad3344a7e13c\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " Nov 01 13:12:26.089011 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.089001 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca\") pod \"bd72f84d-a962-40d7-805b-ad3344a7e13c\" (UID: \"bd72f84d-a962-40d7-805b-ad3344a7e13c\") " Nov 01 13:12:26.089408 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.089369 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca" pod "bd72f84d-a962-40d7-805b-ad3344a7e13c" (UID: "bd72f84d-a962-40d7-805b-ad3344a7e13c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:12:26.090955 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.090930 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca" pod "bd72f84d-a962-40d7-805b-ad3344a7e13c" (UID: "bd72f84d-a962-40d7-805b-ad3344a7e13c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:12:26.091076 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.091001 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk" pod "bd72f84d-a962-40d7-805b-ad3344a7e13c" (UID: "bd72f84d-a962-40d7-805b-ad3344a7e13c"). InnerVolumeSpecName "kube-api-access-s2crk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:12:26.190738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.190698 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:26.190738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.190734 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd72f84d-a962-40d7-805b-ad3344a7e13c-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:26.190738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.190745 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2crk\" (UniqueName: \"kubernetes.io/projected/bd72f84d-a962-40d7-805b-ad3344a7e13c-kube-api-access-s2crk\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:26.800349 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.800322 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" Nov 01 13:12:26.800349 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.800328 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg" event={"ID":"bd72f84d-a962-40d7-805b-ad3344a7e13c","Type":"ContainerDied","Data":"ae0d652fbe19dd72dda1a2730ca68c629ce54fb5657cfa08d196b8468d158bc5"} Nov 01 13:12:26.800572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:26.800375 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae0d652fbe19dd72dda1a2730ca68c629ce54fb5657cfa08d196b8468d158bc5" Nov 01 13:12:30.251490 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:12:30.251448 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:12:33.494353 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.494316 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:12:33.612904 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.612868 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:12:33.789688 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.789611 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls"] Nov 01 13:12:33.790049 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.790036 4484 cpu_manager.go:414] "containerMap: removing container" podUID="bd72f84d-a962-40d7-805b-ad3344a7e13c" containerName="revoke-keys" Nov 01 13:12:33.790098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.790051 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd72f84d-a962-40d7-805b-ad3344a7e13c" containerName="revoke-keys" Nov 01 13:12:33.790143 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.790122 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd72f84d-a962-40d7-805b-ad3344a7e13c" containerName="revoke-keys" Nov 01 13:12:33.793433 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.793411 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.842529 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.842506 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:12:33.853920 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.853889 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.854101 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.853940 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.854101 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.853963 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hv9s\" (UniqueName: \"kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.888909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.888880 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:12:33.891009 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.890988 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls"] Nov 01 13:12:33.891456 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.891439 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:12:33.942895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.942870 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:12:33.955430 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.955401 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.955553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.955452 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.955553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.955480 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hv9s\" (UniqueName: \"kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.956090 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.956071 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:33.957897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:33.957877 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:34.008567 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:34.008536 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hv9s\" (UniqueName: \"kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s\") pod \"maas-api-revoke-sub-3d4156f1-556ls\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:34.106086 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:34.106000 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:34.467603 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:34.467583 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls"] Nov 01 13:12:34.469864 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:12:34.469837 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b20e34a_9370_44c4_91e6_3b2bb7ee4816.slice/crio-ecb2550eabda4caf61756424d432f7791af915b1a131208997794a1761daebc0 WatchSource:0}: Error finding container ecb2550eabda4caf61756424d432f7791af915b1a131208997794a1761daebc0: Status 404 returned error can't find the container with id ecb2550eabda4caf61756424d432f7791af915b1a131208997794a1761daebc0 Nov 01 13:12:34.840215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:34.840182 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" event={"ID":"1b20e34a-9370-44c4-91e6-3b2bb7ee4816","Type":"ContainerStarted","Data":"944be714a255b5b8873e2452e10c946dba92d01a5e14e46cb43ba440b163f2ef"} Nov 01 13:12:34.840735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:34.840237 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" event={"ID":"1b20e34a-9370-44c4-91e6-3b2bb7ee4816","Type":"ContainerStarted","Data":"ecb2550eabda4caf61756424d432f7791af915b1a131208997794a1761daebc0"} Nov 01 13:12:34.948198 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:34.948147 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" podStartSLOduration=1.948134459 podStartE2EDuration="1.948134459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:12:34.945963525 +0000 UTC m=+3024.695906503" watchObservedRunningTime="2026-11-01 13:12:34.948134459 +0000 UTC m=+3024.698077509" Nov 01 13:12:35.414776 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:35.414744 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:12:35.848796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:35.848758 4484 generic.go:356] "Generic (PLEG): container finished" podID="1b20e34a-9370-44c4-91e6-3b2bb7ee4816" containerID="944be714a255b5b8873e2452e10c946dba92d01a5e14e46cb43ba440b163f2ef" exitCode=0 Nov 01 13:12:35.849221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:35.848840 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" event={"ID":"1b20e34a-9370-44c4-91e6-3b2bb7ee4816","Type":"ContainerDied","Data":"944be714a255b5b8873e2452e10c946dba92d01a5e14e46cb43ba440b163f2ef"} Nov 01 13:12:37.220652 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.220618 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:37.229616 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.229583 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca\") pod \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " Nov 01 13:12:37.229742 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.229661 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca\") pod \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " Nov 01 13:12:37.229832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.229812 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s\" (UniqueName: \"kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s\") pod \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\" (UID: \"1b20e34a-9370-44c4-91e6-3b2bb7ee4816\") " Nov 01 13:12:37.230110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.230081 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca" pod "1b20e34a-9370-44c4-91e6-3b2bb7ee4816" (UID: "1b20e34a-9370-44c4-91e6-3b2bb7ee4816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:12:37.231879 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.231854 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s" pod "1b20e34a-9370-44c4-91e6-3b2bb7ee4816" (UID: "1b20e34a-9370-44c4-91e6-3b2bb7ee4816"). InnerVolumeSpecName "kube-api-access-8hv9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:12:37.232069 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.232046 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca" pod "1b20e34a-9370-44c4-91e6-3b2bb7ee4816" (UID: "1b20e34a-9370-44c4-91e6-3b2bb7ee4816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:12:37.330640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.330604 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hv9s\" (UniqueName: \"kubernetes.io/projected/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-kube-api-access-8hv9s\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:37.330640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.330635 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:37.330640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.330647 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20e34a-9370-44c4-91e6-3b2bb7ee4816-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:12:37.876264 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.876223 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" event={"ID":"1b20e34a-9370-44c4-91e6-3b2bb7ee4816","Type":"ContainerDied","Data":"ecb2550eabda4caf61756424d432f7791af915b1a131208997794a1761daebc0"} Nov 01 13:12:37.876493 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.876276 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb2550eabda4caf61756424d432f7791af915b1a131208997794a1761daebc0" Nov 01 13:12:37.876493 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:37.876346 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls" Nov 01 13:12:48.442687 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:48.442653 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:12:55.043555 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:55.043520 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb"] Nov 01 13:12:55.089786 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:55.089753 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2xqkb"] Nov 01 13:12:56.367624 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:56.367592 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89bc765-5939-4636-9c74-e12f4b7588be" path="/var/lib/kubelet/pods/e89bc765-5939-4636-9c74-e12f4b7588be/volumes" Nov 01 13:12:59.134222 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:12:59.134188 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:13:00.250793 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:13:00.250757 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:13:07.428478 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.428443 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls"] Nov 01 13:13:07.474277 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.474246 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-556ls"] Nov 01 13:13:07.540681 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.540651 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8"] Nov 01 13:13:07.541001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.540990 4484 cpu_manager.go:414] "containerMap: removing container" podUID="1b20e34a-9370-44c4-91e6-3b2bb7ee4816" containerName="revoke-keys" Nov 01 13:13:07.541046 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.541003 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b20e34a-9370-44c4-91e6-3b2bb7ee4816" containerName="revoke-keys" Nov 01 13:13:07.541092 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.541083 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b20e34a-9370-44c4-91e6-3b2bb7ee4816" containerName="revoke-keys" Nov 01 13:13:07.544510 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.544492 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.594495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.594457 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:13:07.594704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.594489 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:13:07.594704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.594551 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:13:07.594877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.594712 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:13:07.596758 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.596738 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8"] Nov 01 13:13:07.642527 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.642504 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.642678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.642535 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprvd\" (UniqueName: \"kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.642678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.642566 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.743143 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.743108 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.743143 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.743146 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprvd\" (UniqueName: \"kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.743405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.743184 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.743950 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.743929 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.745576 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.745553 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.796285 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.796250 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprvd\" (UniqueName: \"kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd\") pod \"maas-api-revoke-sub-3d4156f1-cdld8\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:07.857006 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:07.856967 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:08.217851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:08.217726 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8"] Nov 01 13:13:08.220847 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:13:08.220804 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3316e448_0693_4e52_a466_6ca8fe5448cc.slice/crio-0583816818eb7f0905f223bee639448d1b30bb187c0a2f07ea0dea9baf0edd2c WatchSource:0}: Error finding container 0583816818eb7f0905f223bee639448d1b30bb187c0a2f07ea0dea9baf0edd2c: Status 404 returned error can't find the container with id 0583816818eb7f0905f223bee639448d1b30bb187c0a2f07ea0dea9baf0edd2c Nov 01 13:13:08.368504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:08.368470 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b20e34a-9370-44c4-91e6-3b2bb7ee4816" path="/var/lib/kubelet/pods/1b20e34a-9370-44c4-91e6-3b2bb7ee4816/volumes" Nov 01 13:13:09.028122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:09.028081 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" event={"ID":"3316e448-0693-4e52-a466-6ca8fe5448cc","Type":"ContainerStarted","Data":"d1a3a2cf2b4d5dafaf4bb76ffaef725037e8332eb1f42341cc1a2c22191f6340"} Nov 01 13:13:09.028549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:09.028145 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" event={"ID":"3316e448-0693-4e52-a466-6ca8fe5448cc","Type":"ContainerStarted","Data":"0583816818eb7f0905f223bee639448d1b30bb187c0a2f07ea0dea9baf0edd2c"} Nov 01 13:13:09.136546 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:09.136487 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" podStartSLOduration=2.136470992 podStartE2EDuration="2.136470992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:13:09.134287768 +0000 UTC m=+3058.884230746" watchObservedRunningTime="2026-11-01 13:13:09.136470992 +0000 UTC m=+3058.886413970" Nov 01 13:13:10.034753 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:10.034718 4484 generic.go:356] "Generic (PLEG): container finished" podID="3316e448-0693-4e52-a466-6ca8fe5448cc" containerID="d1a3a2cf2b4d5dafaf4bb76ffaef725037e8332eb1f42341cc1a2c22191f6340" exitCode=0 Nov 01 13:13:10.035215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:10.034782 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" event={"ID":"3316e448-0693-4e52-a466-6ca8fe5448cc","Type":"ContainerDied","Data":"d1a3a2cf2b4d5dafaf4bb76ffaef725037e8332eb1f42341cc1a2c22191f6340"} Nov 01 13:13:11.165385 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.165357 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:11.182461 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.182437 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca\") pod \"3316e448-0693-4e52-a466-6ca8fe5448cc\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " Nov 01 13:13:11.182552 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.182490 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca\") pod \"3316e448-0693-4e52-a466-6ca8fe5448cc\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " Nov 01 13:13:11.182606 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.182591 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd\" (UniqueName: \"kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd\") pod \"3316e448-0693-4e52-a466-6ca8fe5448cc\" (UID: \"3316e448-0693-4e52-a466-6ca8fe5448cc\") " Nov 01 13:13:11.182849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.182824 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca" pod "3316e448-0693-4e52-a466-6ca8fe5448cc" (UID: "3316e448-0693-4e52-a466-6ca8fe5448cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:13:11.184503 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.184481 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd" pod "3316e448-0693-4e52-a466-6ca8fe5448cc" (UID: "3316e448-0693-4e52-a466-6ca8fe5448cc"). InnerVolumeSpecName "kube-api-access-nprvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:13:11.184658 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.184638 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca" pod "3316e448-0693-4e52-a466-6ca8fe5448cc" (UID: "3316e448-0693-4e52-a466-6ca8fe5448cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:13:11.285037 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.285007 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:13:11.285037 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.285034 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprvd\" (UniqueName: \"kubernetes.io/projected/3316e448-0693-4e52-a466-6ca8fe5448cc-kube-api-access-nprvd\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:13:11.285210 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:11.285048 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316e448-0693-4e52-a466-6ca8fe5448cc-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:13:12.046747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:12.046718 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" Nov 01 13:13:12.046747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:12.046733 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8" event={"ID":"3316e448-0693-4e52-a466-6ca8fe5448cc","Type":"ContainerDied","Data":"0583816818eb7f0905f223bee639448d1b30bb187c0a2f07ea0dea9baf0edd2c"} Nov 01 13:13:12.046966 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:12.046776 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0583816818eb7f0905f223bee639448d1b30bb187c0a2f07ea0dea9baf0edd2c" Nov 01 13:13:12.174738 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:12.174712 4484 scope.go:122] "RemoveContainer" containerID="9b72f47fbb51c4acc8b80b9640f2b98148530cbb271ce4db4e2a057cc6633740" Nov 01 13:13:17.321244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:17.321201 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:13:22.217567 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:22.217534 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:13:30.250479 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:13:30.250443 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:13:32.527904 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:32.527868 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:13:45.823893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:45.823854 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:13:54.395504 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.395413 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk"] Nov 01 13:13:54.396088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.395955 4484 cpu_manager.go:414] "containerMap: removing container" podUID="3316e448-0693-4e52-a466-6ca8fe5448cc" containerName="revoke-keys" Nov 01 13:13:54.396088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.395976 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3316e448-0693-4e52-a466-6ca8fe5448cc" containerName="revoke-keys" Nov 01 13:13:54.396221 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.396098 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3316e448-0693-4e52-a466-6ca8fe5448cc" containerName="revoke-keys" Nov 01 13:13:54.399321 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.399275 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.473285 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.473253 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.473285 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.473288 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.473506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.473336 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvng\" (UniqueName: \"kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.494767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.494740 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:13:54.494767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.494738 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:13:54.495001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.494746 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:13:54.495001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.494747 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:13:54.496887 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.496864 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk"] Nov 01 13:13:54.574315 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.574274 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvng\" (UniqueName: \"kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.574488 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.574409 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.574488 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.574428 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.575006 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.574986 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.576765 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.576739 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.626930 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.626902 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvng\" (UniqueName: \"kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng\") pod \"maas-api-revoke-sub-e2fc9c5b-8rhqk\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:54.711650 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:54.711585 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:55.074382 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:55.074359 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk"] Nov 01 13:13:55.077369 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:13:55.077341 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda584c9eb_e3a3_4438_8540_4829517114f3.slice/crio-967ec4560ecf8e4d287ac745ea14c1257714fb93c195aa3d23209c19977dd77f WatchSource:0}: Error finding container 967ec4560ecf8e4d287ac745ea14c1257714fb93c195aa3d23209c19977dd77f: Status 404 returned error can't find the container with id 967ec4560ecf8e4d287ac745ea14c1257714fb93c195aa3d23209c19977dd77f Nov 01 13:13:55.251357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:55.251329 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" event={"ID":"a584c9eb-e3a3-4438-8540-4829517114f3","Type":"ContainerStarted","Data":"967ec4560ecf8e4d287ac745ea14c1257714fb93c195aa3d23209c19977dd77f"} Nov 01 13:13:56.257709 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:56.257679 4484 generic.go:356] "Generic (PLEG): container finished" podID="a584c9eb-e3a3-4438-8540-4829517114f3" containerID="06e916d99832e35ea2ddd00085f422e75089eacecaab8b6a1db2827c37646b1f" exitCode=0 Nov 01 13:13:56.258100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:56.257745 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" event={"ID":"a584c9eb-e3a3-4438-8540-4829517114f3","Type":"ContainerDied","Data":"06e916d99832e35ea2ddd00085f422e75089eacecaab8b6a1db2827c37646b1f"} Nov 01 13:13:57.394625 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.394602 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:57.502779 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.502743 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca\") pod \"a584c9eb-e3a3-4438-8540-4829517114f3\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " Nov 01 13:13:57.502952 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.502800 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca\") pod \"a584c9eb-e3a3-4438-8540-4829517114f3\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " Nov 01 13:13:57.502952 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.502904 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng\" (UniqueName: \"kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng\") pod \"a584c9eb-e3a3-4438-8540-4829517114f3\" (UID: \"a584c9eb-e3a3-4438-8540-4829517114f3\") " Nov 01 13:13:57.503168 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.503132 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca" pod "a584c9eb-e3a3-4438-8540-4829517114f3" (UID: "a584c9eb-e3a3-4438-8540-4829517114f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:13:57.505076 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.505045 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca" pod "a584c9eb-e3a3-4438-8540-4829517114f3" (UID: "a584c9eb-e3a3-4438-8540-4829517114f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:13:57.505229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.505206 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng" pod "a584c9eb-e3a3-4438-8540-4829517114f3" (UID: "a584c9eb-e3a3-4438-8540-4829517114f3"). InnerVolumeSpecName "kube-api-access-hvvng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:13:57.603868 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.603810 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvvng\" (UniqueName: \"kubernetes.io/projected/a584c9eb-e3a3-4438-8540-4829517114f3-kube-api-access-hvvng\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:13:57.603868 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.603834 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:13:57.603868 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:57.603845 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a584c9eb-e3a3-4438-8540-4829517114f3-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:13:58.268497 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:58.268470 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" Nov 01 13:13:58.268734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:58.268467 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk" event={"ID":"a584c9eb-e3a3-4438-8540-4829517114f3","Type":"ContainerDied","Data":"967ec4560ecf8e4d287ac745ea14c1257714fb93c195aa3d23209c19977dd77f"} Nov 01 13:13:58.268734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:13:58.268616 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="967ec4560ecf8e4d287ac745ea14c1257714fb93c195aa3d23209c19977dd77f" Nov 01 13:14:00.250403 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:14:00.250356 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:14:01.043189 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:01.043155 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg"] Nov 01 13:14:01.089291 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:01.089261 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-fpfwg"] Nov 01 13:14:02.366936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:02.366902 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164393ce-d1ef-4de3-ad6b-84e4ac874d52" path="/var/lib/kubelet/pods/164393ce-d1ef-4de3-ad6b-84e4ac874d52/volumes" Nov 01 13:14:05.095457 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.095420 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r"] Nov 01 13:14:05.095998 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.095977 4484 cpu_manager.go:414] "containerMap: removing container" podUID="a584c9eb-e3a3-4438-8540-4829517114f3" containerName="revoke-keys" Nov 01 13:14:05.096084 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.096000 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a584c9eb-e3a3-4438-8540-4829517114f3" containerName="revoke-keys" Nov 01 13:14:05.096143 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.096088 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a584c9eb-e3a3-4438-8540-4829517114f3" containerName="revoke-keys" Nov 01 13:14:05.099427 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.099404 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.151928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.151899 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r"] Nov 01 13:14:05.166824 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.166797 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.166969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.166919 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwvqz\" (UniqueName: \"kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.167047 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.167024 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.195427 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.195397 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:14:05.195427 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.195415 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:14:05.195556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.195453 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:14:05.195556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.195462 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:14:05.268049 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.268020 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.268171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.268068 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwvqz\" (UniqueName: \"kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.268171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.268116 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.268670 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.268653 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.270420 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.270399 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.321227 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.321207 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwvqz\" (UniqueName: \"kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz\") pod \"maas-api-revoke-sub-5a29efb3-wvg9r\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.412093 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.412026 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:05.773572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:05.773547 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r"] Nov 01 13:14:05.776544 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:14:05.776514 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853fa2e7_bced_4b32_ab3c_dd8a4208f666.slice/crio-59c147611881d469e7c6098cbf498330c948340403db28b45e036a66a46ce254 WatchSource:0}: Error finding container 59c147611881d469e7c6098cbf498330c948340403db28b45e036a66a46ce254: Status 404 returned error can't find the container with id 59c147611881d469e7c6098cbf498330c948340403db28b45e036a66a46ce254 Nov 01 13:14:06.309432 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:06.309348 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" event={"ID":"853fa2e7-bced-4b32-ab3c-dd8a4208f666","Type":"ContainerStarted","Data":"9b3b7eec5986594a767408ff59f485b904f5460906888988c862ec453f6d37f8"} Nov 01 13:14:06.309432 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:06.309399 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" event={"ID":"853fa2e7-bced-4b32-ab3c-dd8a4208f666","Type":"ContainerStarted","Data":"59c147611881d469e7c6098cbf498330c948340403db28b45e036a66a46ce254"} Nov 01 13:14:06.416949 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:06.416887 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" podStartSLOduration=1.416872112 podStartE2EDuration="1.416872112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:14:06.414657895 +0000 UTC m=+3116.164600873" watchObservedRunningTime="2026-11-01 13:14:06.416872112 +0000 UTC m=+3116.166815090" Nov 01 13:14:07.238945 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:07.238910 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:14:07.315807 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:07.315770 4484 generic.go:356] "Generic (PLEG): container finished" podID="853fa2e7-bced-4b32-ab3c-dd8a4208f666" containerID="9b3b7eec5986594a767408ff59f485b904f5460906888988c862ec453f6d37f8" exitCode=0 Nov 01 13:14:07.316205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:07.315876 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" event={"ID":"853fa2e7-bced-4b32-ab3c-dd8a4208f666","Type":"ContainerDied","Data":"9b3b7eec5986594a767408ff59f485b904f5460906888988c862ec453f6d37f8"} Nov 01 13:14:08.538171 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.538085 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:08.606249 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.606221 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz\" (UniqueName: \"kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz\") pod \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " Nov 01 13:14:08.606594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.606573 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca\") pod \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " Nov 01 13:14:08.606774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.606756 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca\") pod \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\" (UID: \"853fa2e7-bced-4b32-ab3c-dd8a4208f666\") " Nov 01 13:14:08.609801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.609737 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca" pod "853fa2e7-bced-4b32-ab3c-dd8a4208f666" (UID: "853fa2e7-bced-4b32-ab3c-dd8a4208f666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:14:08.612699 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.612676 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca" pod "853fa2e7-bced-4b32-ab3c-dd8a4208f666" (UID: "853fa2e7-bced-4b32-ab3c-dd8a4208f666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:14:08.612822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.612766 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz" pod "853fa2e7-bced-4b32-ab3c-dd8a4208f666" (UID: "853fa2e7-bced-4b32-ab3c-dd8a4208f666"). InnerVolumeSpecName "kube-api-access-gwvqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:14:08.725396 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.725364 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:14:08.725588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.725403 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwvqz\" (UniqueName: \"kubernetes.io/projected/853fa2e7-bced-4b32-ab3c-dd8a4208f666-kube-api-access-gwvqz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:14:08.725588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:08.725419 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/853fa2e7-bced-4b32-ab3c-dd8a4208f666-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:14:09.326672 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:09.326630 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" event={"ID":"853fa2e7-bced-4b32-ab3c-dd8a4208f666","Type":"ContainerDied","Data":"59c147611881d469e7c6098cbf498330c948340403db28b45e036a66a46ce254"} Nov 01 13:14:09.326852 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:09.326694 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59c147611881d469e7c6098cbf498330c948340403db28b45e036a66a46ce254" Nov 01 13:14:09.326852 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:09.326643 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r" Nov 01 13:14:12.194232 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:12.194202 4484 scope.go:122] "RemoveContainer" containerID="218f6461669831af35a35c9975373b9781e5f134bbf13f5cb6660089da42411b" Nov 01 13:14:17.645885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:17.645850 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:14:17.848881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:17.848843 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:14:21.675822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.675788 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8"] Nov 01 13:14:21.721627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.721595 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cdld8"] Nov 01 13:14:21.786640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.786608 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn"] Nov 01 13:14:21.787004 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.786992 4484 cpu_manager.go:414] "containerMap: removing container" podUID="853fa2e7-bced-4b32-ab3c-dd8a4208f666" containerName="revoke-keys" Nov 01 13:14:21.787058 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.787006 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853fa2e7-bced-4b32-ab3c-dd8a4208f666" containerName="revoke-keys" Nov 01 13:14:21.787095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.787089 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="853fa2e7-bced-4b32-ab3c-dd8a4208f666" containerName="revoke-keys" Nov 01 13:14:21.790253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.790234 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:21.840275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.840253 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:14:21.840426 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.840279 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:14:21.840426 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.840387 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:14:21.840564 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.840442 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:14:21.842915 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.842891 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn"] Nov 01 13:14:21.940881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.940781 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:21.940881 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.940840 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh6mz\" (UniqueName: \"kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:21.941081 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:21.940952 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.042332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.042285 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.042506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.042363 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh6mz\" (UniqueName: \"kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.042506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.042410 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.043090 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.043067 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.044722 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.044698 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.095374 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.095353 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh6mz\" (UniqueName: \"kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz\") pod \"maas-api-revoke-sub-3d4156f1-jfwkn\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.103026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.103011 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:22.367055 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.367022 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3316e448-0693-4e52-a466-6ca8fe5448cc" path="/var/lib/kubelet/pods/3316e448-0693-4e52-a466-6ca8fe5448cc/volumes" Nov 01 13:14:22.464111 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:22.464084 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn"] Nov 01 13:14:22.466950 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:14:22.466922 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode006f647_e6d9_4017_8845_a55183e418a1.slice/crio-8e85e36749c54ee3bf17fba46e5daa5243518efffb3c603cbc4424a020ce5b8e WatchSource:0}: Error finding container 8e85e36749c54ee3bf17fba46e5daa5243518efffb3c603cbc4424a020ce5b8e: Status 404 returned error can't find the container with id 8e85e36749c54ee3bf17fba46e5daa5243518efffb3c603cbc4424a020ce5b8e Nov 01 13:14:23.394622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:23.394587 4484 generic.go:356] "Generic (PLEG): container finished" podID="e006f647-e6d9-4017-8845-a55183e418a1" containerID="dd04aa26b8f2cf8cf7eda6bd5ab160c08ec39bd4e920b72a411326f106fe3a3c" exitCode=0 Nov 01 13:14:23.394937 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:23.394630 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" event={"ID":"e006f647-e6d9-4017-8845-a55183e418a1","Type":"ContainerDied","Data":"dd04aa26b8f2cf8cf7eda6bd5ab160c08ec39bd4e920b72a411326f106fe3a3c"} Nov 01 13:14:23.394937 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:23.394691 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" event={"ID":"e006f647-e6d9-4017-8845-a55183e418a1","Type":"ContainerStarted","Data":"8e85e36749c54ee3bf17fba46e5daa5243518efffb3c603cbc4424a020ce5b8e"} Nov 01 13:14:24.528118 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.528096 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:24.665997 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.665917 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca\") pod \"e006f647-e6d9-4017-8845-a55183e418a1\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " Nov 01 13:14:24.665997 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.665967 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca\") pod \"e006f647-e6d9-4017-8845-a55183e418a1\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " Nov 01 13:14:24.666226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.666017 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz\" (UniqueName: \"kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz\") pod \"e006f647-e6d9-4017-8845-a55183e418a1\" (UID: \"e006f647-e6d9-4017-8845-a55183e418a1\") " Nov 01 13:14:24.666326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.666266 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca" pod "e006f647-e6d9-4017-8845-a55183e418a1" (UID: "e006f647-e6d9-4017-8845-a55183e418a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:14:24.668112 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.668089 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca" pod "e006f647-e6d9-4017-8845-a55183e418a1" (UID: "e006f647-e6d9-4017-8845-a55183e418a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:14:24.668209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.668108 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz" pod "e006f647-e6d9-4017-8845-a55183e418a1" (UID: "e006f647-e6d9-4017-8845-a55183e418a1"). InnerVolumeSpecName "kube-api-access-bh6mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:14:24.766813 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.766789 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh6mz\" (UniqueName: \"kubernetes.io/projected/e006f647-e6d9-4017-8845-a55183e418a1-kube-api-access-bh6mz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:14:24.766813 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.766813 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:14:24.766969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:24.766824 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e006f647-e6d9-4017-8845-a55183e418a1-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:14:25.404572 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:25.404541 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" Nov 01 13:14:25.404726 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:25.404574 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn" event={"ID":"e006f647-e6d9-4017-8845-a55183e418a1","Type":"ContainerDied","Data":"8e85e36749c54ee3bf17fba46e5daa5243518efffb3c603cbc4424a020ce5b8e"} Nov 01 13:14:25.404726 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:25.404621 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e85e36749c54ee3bf17fba46e5daa5243518efffb3c603cbc4424a020ce5b8e" Nov 01 13:14:30.250581 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:14:30.250543 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:14:32.043204 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:32.043165 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx"] Nov 01 13:14:32.088972 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:32.088944 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-sshcx"] Nov 01 13:14:32.367353 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:32.367258 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d25f15-805f-4684-b9f0-c6680a97474e" path="/var/lib/kubelet/pods/b8d25f15-805f-4684-b9f0-c6680a97474e/volumes" Nov 01 13:14:36.540956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:36.540925 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:14:45.044346 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:45.044310 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc"] Nov 01 13:14:45.089990 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:45.089957 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-22dtc"] Nov 01 13:14:46.367727 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:46.367684 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a098303-701b-48bc-b79e-784a3886d80c" path="/var/lib/kubelet/pods/1a098303-701b-48bc-b79e-784a3886d80c/volumes" Nov 01 13:14:47.346561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:47.346528 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:14:53.943087 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:53.943051 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw"] Nov 01 13:14:53.943752 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:53.943649 4484 cpu_manager.go:414] "containerMap: removing container" podUID="e006f647-e6d9-4017-8845-a55183e418a1" containerName="revoke-keys" Nov 01 13:14:53.943752 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:53.943670 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e006f647-e6d9-4017-8845-a55183e418a1" containerName="revoke-keys" Nov 01 13:14:53.943885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:53.943766 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e006f647-e6d9-4017-8845-a55183e418a1" containerName="revoke-keys" Nov 01 13:14:53.946981 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:53.946962 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:53.995506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:53.995479 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-483de8-openshift-default-dockercfg-hjhgj\"" Nov 01 13:14:54.029628 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029591 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.029628 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029628 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.029848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029736 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhch\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.029848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029775 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.029848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029795 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.029848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029810 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.030056 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029862 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.030056 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.029956 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.030056 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.030039 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.043801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.043770 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw"] Nov 01 13:14:54.130928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.130892 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.130950 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.130980 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131029 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhch\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131056 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131072 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131088 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131107 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131233 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131385 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131558 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131467 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.131743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131661 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.132022 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.131985 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.133684 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.133665 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.133828 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.133811 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.185926 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.185892 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.187136 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.187104 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhch\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch\") pod \"e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.263164 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.263136 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:54.395872 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.395834 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:14:54.512746 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.512719 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:14:54.677062 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.677011 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw"] Nov 01 13:14:54.679727 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:14:54.679697 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafdc18f_06c8_4ad2_8041_f3dc1d1e8d91.slice/crio-689498e19fcf651894d7270a7fa4c6d3f9f05532fde89be347c6a942d858a2c0 WatchSource:0}: Error finding container 689498e19fcf651894d7270a7fa4c6d3f9f05532fde89be347c6a942d858a2c0: Status 404 returned error can't find the container with id 689498e19fcf651894d7270a7fa4c6d3f9f05532fde89be347c6a942d858a2c0 Nov 01 13:14:54.682167 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.682134 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:14:54.682248 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.682198 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:14:54.682248 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:54.682227 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:14:55.578187 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:55.578153 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" event={"ID":"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91","Type":"ContainerStarted","Data":"4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff"} Nov 01 13:14:55.578608 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:55.578206 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" event={"ID":"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91","Type":"ContainerStarted","Data":"689498e19fcf651894d7270a7fa4c6d3f9f05532fde89be347c6a942d858a2c0"} Nov 01 13:14:55.734206 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:55.734126 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podStartSLOduration=2.734107121 podStartE2EDuration="2.734107121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:14:55.733037285 +0000 UTC m=+3165.482980266" watchObservedRunningTime="2026-11-01 13:14:55.734107121 +0000 UTC m=+3165.484050100" Nov 01 13:14:56.263797 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:56.263755 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:56.268815 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:56.268789 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:56.583419 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:56.583333 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:56.584274 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:56.584252 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:14:58.558755 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:58.558676 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:14:58.721820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:58.721788 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:14:58.848711 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:14:58.848644 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:00.157469 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.157434 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j"] Nov 01 13:15:00.161100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.161078 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.189496 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.189463 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tvm\" (UniqueName: \"kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm\") pod \"maas-api-key-cleanup-29892315-tkt5j\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.189653 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.189632 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892315-tkt5j\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.213763 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.213734 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j"] Nov 01 13:15:00.250973 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:15:00.250927 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:15:00.257137 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.257093 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:15:00.257226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.257099 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:15:00.257226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.257116 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:15:00.290660 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.290629 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tvm\" (UniqueName: \"kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm\") pod \"maas-api-key-cleanup-29892315-tkt5j\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.290821 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.290735 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892315-tkt5j\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.291371 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.291355 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892315-tkt5j\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.344637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.344611 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tvm\" (UniqueName: \"kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm\") pod \"maas-api-key-cleanup-29892315-tkt5j\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.473768 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.473693 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:00.837514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:00.837415 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j"] Nov 01 13:15:00.839827 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:15:00.839798 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f2e92c_e297_4aec_9fad_8cc1bbf54671.slice/crio-84affe139b5a9b187d57d2b763b1bd724a7bae18b93ccce9fe0778992990b618 WatchSource:0}: Error finding container 84affe139b5a9b187d57d2b763b1bd724a7bae18b93ccce9fe0778992990b618: Status 404 returned error can't find the container with id 84affe139b5a9b187d57d2b763b1bd724a7bae18b93ccce9fe0778992990b618 Nov 01 13:15:01.608012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:01.607972 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerStarted","Data":"ea61b5eb498a81ce3f74a8b0e116870f525dd6a4f1b872ddbabdae2e6a2513d9"} Nov 01 13:15:01.608493 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:01.608055 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerStarted","Data":"84affe139b5a9b187d57d2b763b1bd724a7bae18b93ccce9fe0778992990b618"} Nov 01 13:15:01.714805 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:01.714736 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" podStartSLOduration=1.714717099 podStartE2EDuration="1.714717099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:15:01.712659092 +0000 UTC m=+3171.462602069" watchObservedRunningTime="2026-11-01 13:15:01.714717099 +0000 UTC m=+3171.464660081" Nov 01 13:15:01.744502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:01.744426 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:01.910448 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:01.910371 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:02.347158 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.347124 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk"] Nov 01 13:15:02.350826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.350806 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.401574 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.401541 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-483de8\"" Nov 01 13:15:02.401707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.401643 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c4e4f6e3-dockercfg-dfq75\"" Nov 01 13:15:02.417066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.417037 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.417188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.417071 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l\") pod \"payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.449835 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.449812 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk"] Nov 01 13:15:02.489988 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.489964 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc"] Nov 01 13:15:02.493489 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.493472 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.518023 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.518004 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.518130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.518031 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l\") pod \"payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.518730 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.518712 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.546827 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.546805 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc"] Nov 01 13:15:02.571534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.571502 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l\") pod \"payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.613928 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.613855 4484 generic.go:356] "Generic (PLEG): container finished" podID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerID="ea61b5eb498a81ce3f74a8b0e116870f525dd6a4f1b872ddbabdae2e6a2513d9" exitCode=60 Nov 01 13:15:02.614258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.613938 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerDied","Data":"ea61b5eb498a81ce3f74a8b0e116870f525dd6a4f1b872ddbabdae2e6a2513d9"} Nov 01 13:15:02.614258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.614118 4484 scope.go:122] "RemoveContainer" containerID="ea61b5eb498a81ce3f74a8b0e116870f525dd6a4f1b872ddbabdae2e6a2513d9" Nov 01 13:15:02.618851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.618826 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9rx\" (UniqueName: \"kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx\") pod \"payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.618934 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.618865 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.662840 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.662814 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:02.723582 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.723522 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.726080 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.726056 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.726317 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.726269 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9rx\" (UniqueName: \"kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx\") pod \"payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.779635 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.779596 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9rx\" (UniqueName: \"kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx\") pod \"payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:02.806135 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:02.806104 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:03.028103 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.028077 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk"] Nov 01 13:15:03.029776 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:15:03.029745 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb417eb73_fa5b_4be5_888b_3898d766322d.slice/crio-dc25b87c7b4fb95c75cfc594f9319028f9bc5d4031c47ee46b65e50cb28bf30d WatchSource:0}: Error finding container dc25b87c7b4fb95c75cfc594f9319028f9bc5d4031c47ee46b65e50cb28bf30d: Status 404 returned error can't find the container with id dc25b87c7b4fb95c75cfc594f9319028f9bc5d4031c47ee46b65e50cb28bf30d Nov 01 13:15:03.174543 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.174517 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc"] Nov 01 13:15:03.176870 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:15:03.176843 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cdd4be_9ee6_48fe_bcf9_b9e27f90a6a9.slice/crio-cb4cc5dc1e21f5205cdcd5cab7f1dd2a2295a87b6197422b408c576b89e48b4a WatchSource:0}: Error finding container cb4cc5dc1e21f5205cdcd5cab7f1dd2a2295a87b6197422b408c576b89e48b4a: Status 404 returned error can't find the container with id cb4cc5dc1e21f5205cdcd5cab7f1dd2a2295a87b6197422b408c576b89e48b4a Nov 01 13:15:03.620436 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.620405 4484 generic.go:356] "Generic (PLEG): container finished" podID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerID="c79fcbff42aba30fa31d2f93daeb66b7abc77139f60904a1daf94f8561f6cf9e" exitCode=60 Nov 01 13:15:03.620933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.620494 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerDied","Data":"c79fcbff42aba30fa31d2f93daeb66b7abc77139f60904a1daf94f8561f6cf9e"} Nov 01 13:15:03.620933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.620564 4484 scope.go:122] "RemoveContainer" containerID="ea61b5eb498a81ce3f74a8b0e116870f525dd6a4f1b872ddbabdae2e6a2513d9" Nov 01 13:15:03.620933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.620829 4484 scope.go:122] "RemoveContainer" containerID="c79fcbff42aba30fa31d2f93daeb66b7abc77139f60904a1daf94f8561f6cf9e" Nov 01 13:15:03.621113 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:15:03.621091 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892315-tkt5j_odh-ai-gateway-infra(60f2e92c-e297-4aec-9fad-8cc1bbf54671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" Nov 01 13:15:03.622271 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.622247 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" event={"ID":"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9","Type":"ContainerStarted","Data":"29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a"} Nov 01 13:15:03.622435 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.622318 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" event={"ID":"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9","Type":"ContainerStarted","Data":"cb4cc5dc1e21f5205cdcd5cab7f1dd2a2295a87b6197422b408c576b89e48b4a"} Nov 01 13:15:03.622435 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.622365 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:03.623807 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.623777 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" event={"ID":"b417eb73-fa5b-4be5-888b-3898d766322d","Type":"ContainerStarted","Data":"533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e"} Nov 01 13:15:03.623903 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.623826 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" event={"ID":"b417eb73-fa5b-4be5-888b-3898d766322d","Type":"ContainerStarted","Data":"dc25b87c7b4fb95c75cfc594f9319028f9bc5d4031c47ee46b65e50cb28bf30d"} Nov 01 13:15:03.623903 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.623866 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:03.833950 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.833902 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" podStartSLOduration=1.833888128 podStartE2EDuration="1.833888128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:15:03.832553517 +0000 UTC m=+3173.582496507" watchObservedRunningTime="2026-11-01 13:15:03.833888128 +0000 UTC m=+3173.583831106" Nov 01 13:15:03.939632 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:03.939540 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" podStartSLOduration=1.939527987 podStartE2EDuration="1.939527987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:15:03.93812778 +0000 UTC m=+3173.688070759" watchObservedRunningTime="2026-11-01 13:15:03.939527987 +0000 UTC m=+3173.689470964" Nov 01 13:15:04.460068 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:04.460038 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:04.590502 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:04.590468 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:04.630849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:04.630818 4484 scope.go:122] "RemoveContainer" containerID="c79fcbff42aba30fa31d2f93daeb66b7abc77139f60904a1daf94f8561f6cf9e" Nov 01 13:15:04.631229 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:15:04.631095 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892315-tkt5j_odh-ai-gateway-infra(60f2e92c-e297-4aec-9fad-8cc1bbf54671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" Nov 01 13:15:04.705689 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:04.705654 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:09.109290 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:09.109254 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:12.213237 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:12.213197 4484 scope.go:122] "RemoveContainer" containerID="f653b79c23b9ed5d4285406ffc49e2c46fe467431b58e29fbed32cf44cebfc38" Nov 01 13:15:12.333427 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:12.333390 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:15:13.628403 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:13.628368 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:14.633889 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:14.633863 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:15:14.634284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:14.633926 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:15:15.361977 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:15.361942 4484 scope.go:122] "RemoveContainer" containerID="c79fcbff42aba30fa31d2f93daeb66b7abc77139f60904a1daf94f8561f6cf9e" Nov 01 13:15:15.690374 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:15.690346 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerStarted","Data":"882a67d4e2ba268ba3674ef296cafad273213f60ca42b544917cd8cc6b9f8c0b"} Nov 01 13:15:16.697195 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:16.697104 4484 generic.go:356] "Generic (PLEG): container finished" podID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerID="882a67d4e2ba268ba3674ef296cafad273213f60ca42b544917cd8cc6b9f8c0b" exitCode=60 Nov 01 13:15:16.697195 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:16.697176 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerDied","Data":"882a67d4e2ba268ba3674ef296cafad273213f60ca42b544917cd8cc6b9f8c0b"} Nov 01 13:15:16.697698 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:16.697238 4484 scope.go:122] "RemoveContainer" containerID="c79fcbff42aba30fa31d2f93daeb66b7abc77139f60904a1daf94f8561f6cf9e" Nov 01 13:15:16.697698 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:16.697593 4484 scope.go:122] "RemoveContainer" containerID="882a67d4e2ba268ba3674ef296cafad273213f60ca42b544917cd8cc6b9f8c0b" Nov 01 13:15:16.697855 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:15:16.697837 4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892315-tkt5j_odh-ai-gateway-infra(60f2e92c-e297-4aec-9fad-8cc1bbf54671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" Nov 01 13:15:16.810122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:16.810091 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j"] Nov 01 13:15:17.877980 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:17.877945 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:17.909320 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:17.909267 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm\" (UniqueName: \"kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm\") pod \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " Nov 01 13:15:17.909464 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:17.909445 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca\") pod \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\" (UID: \"60f2e92c-e297-4aec-9fad-8cc1bbf54671\") " Nov 01 13:15:17.909796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:17.909771 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca" pod "60f2e92c-e297-4aec-9fad-8cc1bbf54671" (UID: "60f2e92c-e297-4aec-9fad-8cc1bbf54671"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:15:17.911375 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:17.911357 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm" pod "60f2e92c-e297-4aec-9fad-8cc1bbf54671" (UID: "60f2e92c-e297-4aec-9fad-8cc1bbf54671"). InnerVolumeSpecName "kube-api-access-d8tvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:15:17.966380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:17.966161 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:18.010175 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.010137 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tvm\" (UniqueName: \"kubernetes.io/projected/60f2e92c-e297-4aec-9fad-8cc1bbf54671-kube-api-access-d8tvm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:15:18.010175 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.010167 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e92c-e297-4aec-9fad-8cc1bbf54671-maas-api-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:15:18.145015 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.144976 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:18.710447 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.710374 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" Nov 01 13:15:18.710610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.710371 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j" event={"ID":"60f2e92c-e297-4aec-9fad-8cc1bbf54671","Type":"ContainerDied","Data":"84affe139b5a9b187d57d2b763b1bd724a7bae18b93ccce9fe0778992990b618"} Nov 01 13:15:18.710610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.710511 4484 scope.go:122] "RemoveContainer" containerID="882a67d4e2ba268ba3674ef296cafad273213f60ca42b544917cd8cc6b9f8c0b" Nov 01 13:15:18.861810 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.861772 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j"] Nov 01 13:15:18.866683 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:18.866654 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892315-tkt5j"] Nov 01 13:15:19.058948 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:19.058907 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j"] Nov 01 13:15:19.062606 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:19.062582 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-86g4j"] Nov 01 13:15:20.368924 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:20.368889 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" path="/var/lib/kubelet/pods/60f2e92c-e297-4aec-9fad-8cc1bbf54671/volumes" Nov 01 13:15:20.369325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:20.369313 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5720cd-0966-428b-b26b-666376855307" path="/var/lib/kubelet/pods/9e5720cd-0966-428b-b26b-666376855307/volumes" Nov 01 13:15:21.729455 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:21.729417 4484 generic.go:356] "Generic (PLEG): container finished" podID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerID="4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff" exitCode=0 Nov 01 13:15:21.729917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:21.729485 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" event={"ID":"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91","Type":"ContainerDied","Data":"4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff"} Nov 01 13:15:21.729917 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:21.729865 4484 scope.go:122] "RemoveContainer" containerID="4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff" Nov 01 13:15:21.731101 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:21.731060 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:15:21.731210 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:21.731146 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:15:21.731210 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:21.731184 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:15:22.737889 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:22.737854 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" event={"ID":"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91","Type":"ContainerStarted","Data":"0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2"} Nov 01 13:15:22.921735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:22.921695 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:23.264322 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:23.263673 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:15:23.265648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:23.265620 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 13:15:23.265858 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:23.265831 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 13:15:23.423003 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:23.422969 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:24.027098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:24.027064 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:24.167624 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:24.167588 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:24.272327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:24.271854 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 13:15:24.272327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:24.271944 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 13:15:24.301769 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:24.301651 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:25.266326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:25.263766 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 13:15:25.266326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:25.263873 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 13:15:26.263817 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:26.263776 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 13:15:26.264149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:26.264119 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 13:15:26.584222 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:26.584135 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:15:26.690607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:26.690576 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:27.264325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:27.263730 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 13:15:27.264325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:27.263796 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 13:15:27.516055 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:27.515948 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:28.267756 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:28.267724 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:15:28.268754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:28.268738 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:15:30.042956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:30.042915 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh"] Nov 01 13:15:30.088548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:30.088515 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-gbzxh"] Nov 01 13:15:30.250792 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:15:30.250746 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:15:30.368321 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:30.368235 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3366700-e1fa-4720-ac8b-126f9d9efc85" path="/var/lib/kubelet/pods/b3366700-e1fa-4720-ac8b-126f9d9efc85/volumes" Nov 01 13:15:31.269008 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:31.268972 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:15:32.030136 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:32.030107 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:36.457488 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:36.457453 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:40.894968 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:40.894934 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:45.297028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:45.296993 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:49.702942 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:49.702911 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:54.102885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:54.102850 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:15:57.246451 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:57.246416 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:15:58.611162 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:15:58.611128 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:00.250092 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:16:00.250057 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:16:03.002607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:03.002567 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:04.044795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:04.044759 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs"] Nov 01 13:16:04.091045 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:04.091012 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jknxs"] Nov 01 13:16:04.368181 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:04.368104 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60da4116-4958-49e6-a0d9-bc5a52fcf17e" path="/var/lib/kubelet/pods/60da4116-4958-49e6-a0d9-bc5a52fcf17e/volumes" Nov 01 13:16:07.415936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:07.415888 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:11.537534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537498 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c"] Nov 01 13:16:11.537927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537883 4484 cpu_manager.go:414] "containerMap: removing container" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.537927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537893 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.537927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537906 4484 cpu_manager.go:414] "containerMap: removing container" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.537927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537911 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.537927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537923 4484 cpu_manager.go:414] "containerMap: removing container" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.537927 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537929 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.538130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537984 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.538130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.537994 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:11.541497 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.541479 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.619610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.619564 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.619796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.619684 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6kg\" (UniqueName: \"kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.619880 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.619853 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.638001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.637967 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:16:11.638001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.637967 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:16:11.638001 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.637971 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:16:11.638263 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.638127 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:16:11.640122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.640102 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c"] Nov 01 13:16:11.720792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.720759 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.720792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.720804 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.721064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.720834 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6kg\" (UniqueName: \"kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.721532 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.721508 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.723182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.723160 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.773773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.773746 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6kg\" (UniqueName: \"kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg\") pod \"maas-api-revoke-sub-ccab7d7c-dvt4c\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:11.808018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.807931 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:11.853838 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:11.853815 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:12.234183 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:12.234152 4484 scope.go:122] "RemoveContainer" containerID="27b65ff6ef83b5e3f3607de9554684550a1ec09eff45cffb1d8cae0bcc76ec20" Nov 01 13:16:12.244165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:12.244142 4484 scope.go:122] "RemoveContainer" containerID="01afc055cfdf1a4a0ed9a1585ae64be4fd31041775da17661f15ae4840d8ef82" Nov 01 13:16:12.247430 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:12.247409 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c"] Nov 01 13:16:12.249884 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:16:12.249854 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4385c0_e814_460b_b898_ae4ac14632d0.slice/crio-acf2d8857979b30c178be1019054698c837cceacacfb722f4bf0287b3f6cf100 WatchSource:0}: Error finding container acf2d8857979b30c178be1019054698c837cceacacfb722f4bf0287b3f6cf100: Status 404 returned error can't find the container with id acf2d8857979b30c178be1019054698c837cceacacfb722f4bf0287b3f6cf100 Nov 01 13:16:13.016402 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:13.016361 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" event={"ID":"fc4385c0-e814-460b-b898-ae4ac14632d0","Type":"ContainerStarted","Data":"8d5ab910e0170f987c59f8b0f472b087a77ff27031be042d5380156c6221c45d"} Nov 01 13:16:13.016796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:13.016428 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" event={"ID":"fc4385c0-e814-460b-b898-ae4ac14632d0","Type":"ContainerStarted","Data":"acf2d8857979b30c178be1019054698c837cceacacfb722f4bf0287b3f6cf100"} Nov 01 13:16:13.123155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:13.123101 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" podStartSLOduration=2.123087884 podStartE2EDuration="2.123087884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:16:13.121784068 +0000 UTC m=+3242.871727058" watchObservedRunningTime="2026-11-01 13:16:13.123087884 +0000 UTC m=+3242.873030861" Nov 01 13:16:14.022469 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:14.022434 4484 generic.go:356] "Generic (PLEG): container finished" podID="fc4385c0-e814-460b-b898-ae4ac14632d0" containerID="8d5ab910e0170f987c59f8b0f472b087a77ff27031be042d5380156c6221c45d" exitCode=0 Nov 01 13:16:14.022878 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:14.022519 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" event={"ID":"fc4385c0-e814-460b-b898-ae4ac14632d0","Type":"ContainerDied","Data":"8d5ab910e0170f987c59f8b0f472b087a77ff27031be042d5380156c6221c45d"} Nov 01 13:16:15.161153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.161127 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:15.247893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.247862 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca\") pod \"fc4385c0-e814-460b-b898-ae4ac14632d0\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " Nov 01 13:16:15.248035 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.247943 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca\") pod \"fc4385c0-e814-460b-b898-ae4ac14632d0\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " Nov 01 13:16:15.248035 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.247983 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg\" (UniqueName: \"kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg\") pod \"fc4385c0-e814-460b-b898-ae4ac14632d0\" (UID: \"fc4385c0-e814-460b-b898-ae4ac14632d0\") " Nov 01 13:16:15.248288 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.248261 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca" pod "fc4385c0-e814-460b-b898-ae4ac14632d0" (UID: "fc4385c0-e814-460b-b898-ae4ac14632d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:16:15.250005 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.249983 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg" pod "fc4385c0-e814-460b-b898-ae4ac14632d0" (UID: "fc4385c0-e814-460b-b898-ae4ac14632d0"). InnerVolumeSpecName "kube-api-access-pz6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:15.250005 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.249994 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca" pod "fc4385c0-e814-460b-b898-ae4ac14632d0" (UID: "fc4385c0-e814-460b-b898-ae4ac14632d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:16:15.349322 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.349224 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz6kg\" (UniqueName: \"kubernetes.io/projected/fc4385c0-e814-460b-b898-ae4ac14632d0-kube-api-access-pz6kg\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:15.349322 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.349250 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:15.349322 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:15.349261 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4385c0-e814-460b-b898-ae4ac14632d0-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:16.033953 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:16.033926 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" Nov 01 13:16:16.034125 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:16.033922 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c" event={"ID":"fc4385c0-e814-460b-b898-ae4ac14632d0","Type":"ContainerDied","Data":"acf2d8857979b30c178be1019054698c837cceacacfb722f4bf0287b3f6cf100"} Nov 01 13:16:16.034125 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:16.034054 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf2d8857979b30c178be1019054698c837cceacacfb722f4bf0287b3f6cf100" Nov 01 13:16:16.315070 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:16.314997 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:18.296190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:18.296157 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:20.696520 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:20.696487 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:23.074724 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.074686 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:23.594145 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.594105 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs"] Nov 01 13:16:23.594603 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.594585 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fc4385c0-e814-460b-b898-ae4ac14632d0" containerName="revoke-keys" Nov 01 13:16:23.594714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.594605 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4385c0-e814-460b-b898-ae4ac14632d0" containerName="revoke-keys" Nov 01 13:16:23.594714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.594706 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f2e92c-e297-4aec-9fad-8cc1bbf54671" containerName="cleanup" Nov 01 13:16:23.594831 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.594723 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4385c0-e814-460b-b898-ae4ac14632d0" containerName="revoke-keys" Nov 01 13:16:23.597964 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.597944 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.629836 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.629814 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vbk\" (UniqueName: \"kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.629950 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.629859 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.629998 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.629948 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.650349 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.650328 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs"] Nov 01 13:16:23.693997 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.693970 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:16:23.694110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.693999 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:16:23.694110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.694022 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:16:23.694110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.694048 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:16:23.730591 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.730564 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vbk\" (UniqueName: \"kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.730696 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.730618 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.730696 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.730665 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.731283 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.731256 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.732986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.732966 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.783999 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.783979 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vbk\" (UniqueName: \"kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:23.910464 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:23.910399 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:24.275540 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:24.275513 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs"] Nov 01 13:16:24.278562 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:16:24.278528 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c19738_c3be_4345_b2ba_f73dbc4f585b.slice/crio-84f9a56d978769d27e32584ac9ac0160af34c2c017d57e2cfa6c0a4f25fa9efb WatchSource:0}: Error finding container 84f9a56d978769d27e32584ac9ac0160af34c2c017d57e2cfa6c0a4f25fa9efb: Status 404 returned error can't find the container with id 84f9a56d978769d27e32584ac9ac0160af34c2c017d57e2cfa6c0a4f25fa9efb Nov 01 13:16:25.079093 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:25.079059 4484 generic.go:356] "Generic (PLEG): container finished" podID="d1c19738-c3be-4345-b2ba-f73dbc4f585b" containerID="111c5cfa38477a0531915e5905d1586dedea5c85b98c1b416b552a09f18f268a" exitCode=0 Nov 01 13:16:25.079258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:25.079113 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" event={"ID":"d1c19738-c3be-4345-b2ba-f73dbc4f585b","Type":"ContainerDied","Data":"111c5cfa38477a0531915e5905d1586dedea5c85b98c1b416b552a09f18f268a"} Nov 01 13:16:25.079258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:25.079153 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" event={"ID":"d1c19738-c3be-4345-b2ba-f73dbc4f585b","Type":"ContainerStarted","Data":"84f9a56d978769d27e32584ac9ac0160af34c2c017d57e2cfa6c0a4f25fa9efb"} Nov 01 13:16:25.344806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:25.344722 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:26.154968 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.154935 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:16:26.245699 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.245672 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:26.357561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.357478 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca\") pod \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " Nov 01 13:16:26.357561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.357517 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca\") pod \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " Nov 01 13:16:26.358095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.357569 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk\" (UniqueName: \"kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk\") pod \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\" (UID: \"d1c19738-c3be-4345-b2ba-f73dbc4f585b\") " Nov 01 13:16:26.358095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.357891 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca" pod "d1c19738-c3be-4345-b2ba-f73dbc4f585b" (UID: "d1c19738-c3be-4345-b2ba-f73dbc4f585b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:16:26.359901 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.359874 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca" pod "d1c19738-c3be-4345-b2ba-f73dbc4f585b" (UID: "d1c19738-c3be-4345-b2ba-f73dbc4f585b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:16:26.360038 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.360011 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk" pod "d1c19738-c3be-4345-b2ba-f73dbc4f585b" (UID: "d1c19738-c3be-4345-b2ba-f73dbc4f585b"). InnerVolumeSpecName "kube-api-access-44vbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:26.459514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.459472 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vbk\" (UniqueName: \"kubernetes.io/projected/d1c19738-c3be-4345-b2ba-f73dbc4f585b-kube-api-access-44vbk\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:26.459514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.459509 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:26.459514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:26.459520 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c19738-c3be-4345-b2ba-f73dbc4f585b-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:27.090013 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:27.089982 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" Nov 01 13:16:27.090195 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:27.090011 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs" event={"ID":"d1c19738-c3be-4345-b2ba-f73dbc4f585b","Type":"ContainerDied","Data":"84f9a56d978769d27e32584ac9ac0160af34c2c017d57e2cfa6c0a4f25fa9efb"} Nov 01 13:16:27.090195 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:27.090056 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f9a56d978769d27e32584ac9ac0160af34c2c017d57e2cfa6c0a4f25fa9efb" Nov 01 13:16:28.143321 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:28.143262 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:30.095106 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:30.095069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:30.252566 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:16:30.252524 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:16:32.041616 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:32.041580 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4"] Nov 01 13:16:32.087158 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:32.087131 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kt2p4"] Nov 01 13:16:32.367729 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:32.367652 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba370f5e-c6ce-4916-90a3-b5e8ec0bff49" path="/var/lib/kubelet/pods/ba370f5e-c6ce-4916-90a3-b5e8ec0bff49/volumes" Nov 01 13:16:33.980415 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:33.980374 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:34.312274 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:34.312193 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk"] Nov 01 13:16:34.360953 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:34.360917 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc"] Nov 01 13:16:34.441181 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:34.441112 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:34.557450 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:34.557410 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:37.052781 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:37.052746 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:16:38.690935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:38.690902 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:38.936898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:38.936865 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:39.252801 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:39.252763 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs"] Nov 01 13:16:39.298204 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:39.298168 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-483de8-t27bs"] Nov 01 13:16:39.313217 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:39.313181 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" containerName="payload-processing" containerID="cri-o://533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e" gracePeriod=25 Nov 01 13:16:39.361879 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:39.361831 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerName="payload-pre-processing" containerID="cri-o://29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a" gracePeriod=25 Nov 01 13:16:39.672358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:39.672256 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:40.368019 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:40.367984 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c19738-c3be-4345-b2ba-f73dbc4f585b" path="/var/lib/kubelet/pods/d1c19738-c3be-4345-b2ba-f73dbc4f585b/volumes" Nov 01 13:16:44.631883 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:44.631847 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:16:44.632288 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:44.631904 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:16:45.366092 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:45.366055 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:45.512375 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:45.512296 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:45.752140 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:45.752109 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw"] Nov 01 13:16:45.752537 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:45.752337 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" containerID="cri-o://0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2" gracePeriod=30 Nov 01 13:16:45.763871 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:45.763845 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:16:47.205407 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.205371 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn"] Nov 01 13:16:47.251702 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.251670 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfwkn"] Nov 01 13:16:47.317679 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.317646 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk"] Nov 01 13:16:47.318101 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.318086 4484 cpu_manager.go:414] "containerMap: removing container" podUID="d1c19738-c3be-4345-b2ba-f73dbc4f585b" containerName="revoke-keys" Nov 01 13:16:47.318146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.318103 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c19738-c3be-4345-b2ba-f73dbc4f585b" containerName="revoke-keys" Nov 01 13:16:47.318183 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.318164 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c19738-c3be-4345-b2ba-f73dbc4f585b" containerName="revoke-keys" Nov 01 13:16:47.321203 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.321186 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.372399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.372370 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:16:47.372399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.372386 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:16:47.372804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.372407 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:16:47.372804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.372563 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:16:47.375278 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.375216 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk"] Nov 01 13:16:47.467775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.467690 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvv9\" (UniqueName: \"kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.467775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.467736 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.467971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.467878 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.568588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.568553 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.568783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.568642 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvv9\" (UniqueName: \"kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.568783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.568669 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.569258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.569235 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.570898 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.570879 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.621418 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.621395 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvv9\" (UniqueName: \"kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9\") pod \"maas-api-revoke-sub-3d4156f1-66cbk\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.633794 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.633770 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:47.998977 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:47.998899 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk"] Nov 01 13:16:48.005619 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:16:48.005588 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479a9400_687c_499c_bb8a_16c129d665a6.slice/crio-ca3a9bfc24095ffc7c1e2e0b9b9e68d44e22594b4f3a4e2c3452111bbdac4dce WatchSource:0}: Error finding container ca3a9bfc24095ffc7c1e2e0b9b9e68d44e22594b4f3a4e2c3452111bbdac4dce: Status 404 returned error can't find the container with id ca3a9bfc24095ffc7c1e2e0b9b9e68d44e22594b4f3a4e2c3452111bbdac4dce Nov 01 13:16:48.202342 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:48.202294 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" event={"ID":"479a9400-687c-499c-bb8a-16c129d665a6","Type":"ContainerStarted","Data":"ca3a9bfc24095ffc7c1e2e0b9b9e68d44e22594b4f3a4e2c3452111bbdac4dce"} Nov 01 13:16:48.368129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:48.368092 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e006f647-e6d9-4017-8845-a55183e418a1" path="/var/lib/kubelet/pods/e006f647-e6d9-4017-8845-a55183e418a1/volumes" Nov 01 13:16:49.209038 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:49.208955 4484 generic.go:356] "Generic (PLEG): container finished" podID="479a9400-687c-499c-bb8a-16c129d665a6" containerID="b17d51e68115716ffca529d30c7e0650be8b84818292adcfe1cb689b7443cf5a" exitCode=0 Nov 01 13:16:49.209194 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:49.209037 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" event={"ID":"479a9400-687c-499c-bb8a-16c129d665a6","Type":"ContainerDied","Data":"b17d51e68115716ffca529d30c7e0650be8b84818292adcfe1cb689b7443cf5a"} Nov 01 13:16:50.349105 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.349076 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:50.497109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.497079 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9\" (UniqueName: \"kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9\") pod \"479a9400-687c-499c-bb8a-16c129d665a6\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " Nov 01 13:16:50.497325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.497121 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca\") pod \"479a9400-687c-499c-bb8a-16c129d665a6\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " Nov 01 13:16:50.497325 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.497139 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca\") pod \"479a9400-687c-499c-bb8a-16c129d665a6\" (UID: \"479a9400-687c-499c-bb8a-16c129d665a6\") " Nov 01 13:16:50.497548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.497525 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca" pod "479a9400-687c-499c-bb8a-16c129d665a6" (UID: "479a9400-687c-499c-bb8a-16c129d665a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:16:50.499252 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.499224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca" pod "479a9400-687c-499c-bb8a-16c129d665a6" (UID: "479a9400-687c-499c-bb8a-16c129d665a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:16:50.499358 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.499334 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9" pod "479a9400-687c-499c-bb8a-16c129d665a6" (UID: "479a9400-687c-499c-bb8a-16c129d665a6"). InnerVolumeSpecName "kube-api-access-bvvv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:50.598562 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.598522 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvvv9\" (UniqueName: \"kubernetes.io/projected/479a9400-687c-499c-bb8a-16c129d665a6-kube-api-access-bvvv9\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:50.598562 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.598556 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:50.598562 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.598568 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479a9400-687c-499c-bb8a-16c129d665a6-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:50.986606 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:50.986581 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:16:51.103450 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103357 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103450 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103420 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103452 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103501 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103540 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103573 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103589 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103610 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.103717 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103663 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch\") pod \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\" (UID: \"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91\") " Nov 01 13:16:51.104122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.103935 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:16:51.104122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.104011 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:16:51.104359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.104248 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:16:51.104491 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.104399 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:16:51.104491 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.104463 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:16:51.106105 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.106082 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:51.106199 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.106102 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "kube-api-access-jxhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:51.106384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.106361 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:16:51.106481 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.106394 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo" pod "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" (UID: "dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:16:51.204832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204788 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.204832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204823 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.204832 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204839 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.205095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204852 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.205095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204864 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhch\" (UniqueName: \"kubernetes.io/projected/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-kube-api-access-jxhch\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.205095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204877 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.205095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204889 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.205095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204900 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.205095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.204912 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:51.221014 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.220982 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" Nov 01 13:16:51.221188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.221021 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk" event={"ID":"479a9400-687c-499c-bb8a-16c129d665a6","Type":"ContainerDied","Data":"ca3a9bfc24095ffc7c1e2e0b9b9e68d44e22594b4f3a4e2c3452111bbdac4dce"} Nov 01 13:16:51.221188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.221084 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3a9bfc24095ffc7c1e2e0b9b9e68d44e22594b4f3a4e2c3452111bbdac4dce" Nov 01 13:16:51.222796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.222771 4484 generic.go:356] "Generic (PLEG): container finished" podID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerID="0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2" exitCode=0 Nov 01 13:16:51.222948 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.222820 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" Nov 01 13:16:51.222948 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.222828 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" event={"ID":"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91","Type":"ContainerDied","Data":"0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2"} Nov 01 13:16:51.222948 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.222872 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw" event={"ID":"dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91","Type":"ContainerDied","Data":"689498e19fcf651894d7270a7fa4c6d3f9f05532fde89be347c6a942d858a2c0"} Nov 01 13:16:51.222948 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.222900 4484 scope.go:122] "RemoveContainer" containerID="0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2" Nov 01 13:16:51.232583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.232558 4484 scope.go:122] "RemoveContainer" containerID="4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff" Nov 01 13:16:51.241332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.241292 4484 scope.go:122] "RemoveContainer" containerID="0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2" Nov 01 13:16:51.241608 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:16:51.241587 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2\": container with ID starting with 0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2 not found: ID does not exist" containerID="0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2" Nov 01 13:16:51.241697 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.241623 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2"} err="failed to get container status \"0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2\": rpc error: code = NotFound desc = could not find container \"0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2\": container with ID starting with 0fabcc1153bc13eeb28964ff7a67178e1ddce6277b21211410701edd01f4eba2 not found: ID does not exist" Nov 01 13:16:51.241697 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.241653 4484 scope.go:122] "RemoveContainer" containerID="4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff" Nov 01 13:16:51.241932 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:16:51.241911 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff\": container with ID starting with 4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff not found: ID does not exist" containerID="4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff" Nov 01 13:16:51.241969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.241939 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff"} err="failed to get container status \"4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff\": rpc error: code = NotFound desc = could not find container \"4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff\": container with ID starting with 4ca8cc2e5459b24908437bf48e87f23dcd15dd716afeda4a40cf5c2bd4f7dbff not found: ID does not exist" Nov 01 13:16:51.424812 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.424730 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw"] Nov 01 13:16:51.429640 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:51.429618 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-483de8-openshift-default-846d7b9cb9l6whw"] Nov 01 13:16:52.367337 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:52.367282 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" path="/var/lib/kubelet/pods/dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91/volumes" Nov 01 13:16:54.632945 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:54.632864 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:16:54.633370 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:54.632863 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:16:59.604340 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.604314 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:16:59.633209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.633188 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:16:59.676692 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.676667 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume\") pod \"b417eb73-fa5b-4be5-888b-3898d766322d\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " Nov 01 13:16:59.676820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.676762 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx\" (UniqueName: \"kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx\") pod \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " Nov 01 13:16:59.676820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.676785 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume\") pod \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\" (UID: \"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9\") " Nov 01 13:16:59.676949 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.676816 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l\") pod \"b417eb73-fa5b-4be5-888b-3898d766322d\" (UID: \"b417eb73-fa5b-4be5-888b-3898d766322d\") " Nov 01 13:16:59.677152 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.677126 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume" pod "b417eb73-fa5b-4be5-888b-3898d766322d" (UID: "b417eb73-fa5b-4be5-888b-3898d766322d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:16:59.677209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.677153 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume" pod "63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" (UID: "63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:16:59.678849 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.678831 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx" pod "63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" (UID: "63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9"). InnerVolumeSpecName "kube-api-access-4t9rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:59.678918 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.678865 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l" pod "b417eb73-fa5b-4be5-888b-3898d766322d" (UID: "b417eb73-fa5b-4be5-888b-3898d766322d"). InnerVolumeSpecName "kube-api-access-4k84l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:16:59.777792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.777763 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b417eb73-fa5b-4be5-888b-3898d766322d-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:59.777792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.777787 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9rx\" (UniqueName: \"kubernetes.io/projected/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-kube-api-access-4t9rx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:59.777792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.777796 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:16:59.777993 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:16:59.777806 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k84l\" (UniqueName: \"kubernetes.io/projected/b417eb73-fa5b-4be5-888b-3898d766322d-kube-api-access-4k84l\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:17:00.252027 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:17:00.251992 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:17:00.266594 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.266559 4484 generic.go:356] "Generic (PLEG): container finished" podID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerID="29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a" exitCode=0 Nov 01 13:17:00.266749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.266603 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" Nov 01 13:17:00.266749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.266640 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" event={"ID":"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9","Type":"ContainerDied","Data":"29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a"} Nov 01 13:17:00.266749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.266690 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc" event={"ID":"63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9","Type":"ContainerDied","Data":"cb4cc5dc1e21f5205cdcd5cab7f1dd2a2295a87b6197422b408c576b89e48b4a"} Nov 01 13:17:00.266749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.266712 4484 scope.go:122] "RemoveContainer" containerID="29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a" Nov 01 13:17:00.268009 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.267988 4484 generic.go:356] "Generic (PLEG): container finished" podID="b417eb73-fa5b-4be5-888b-3898d766322d" containerID="533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e" exitCode=0 Nov 01 13:17:00.268122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.268035 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" Nov 01 13:17:00.268122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.268074 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" event={"ID":"b417eb73-fa5b-4be5-888b-3898d766322d","Type":"ContainerDied","Data":"533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e"} Nov 01 13:17:00.268122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.268107 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk" event={"ID":"b417eb73-fa5b-4be5-888b-3898d766322d","Type":"ContainerDied","Data":"dc25b87c7b4fb95c75cfc594f9319028f9bc5d4031c47ee46b65e50cb28bf30d"} Nov 01 13:17:00.277165 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.277150 4484 scope.go:122] "RemoveContainer" containerID="29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a" Nov 01 13:17:00.277450 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:17:00.277431 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a\": container with ID starting with 29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a not found: ID does not exist" containerID="29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a" Nov 01 13:17:00.277556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.277457 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a"} err="failed to get container status \"29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a\": rpc error: code = NotFound desc = could not find container \"29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a\": container with ID starting with 29820e7d09928529c0e7a48bf7e982e8fb61cb3d7dc069e1abdc5d5abbd3535a not found: ID does not exist" Nov 01 13:17:00.277556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.277476 4484 scope.go:122] "RemoveContainer" containerID="533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e" Nov 01 13:17:00.285804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.285788 4484 scope.go:122] "RemoveContainer" containerID="533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e" Nov 01 13:17:00.286047 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:17:00.286029 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e\": container with ID starting with 533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e not found: ID does not exist" containerID="533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e" Nov 01 13:17:00.286120 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.286051 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e"} err="failed to get container status \"533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e\": rpc error: code = NotFound desc = could not find container \"533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e\": container with ID starting with 533c398427c6c66fc67ef4d9e31ffc63b5afb29377ef816f085649a156ba592e not found: ID does not exist" Nov 01 13:17:00.420410 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.420382 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk"] Nov 01 13:17:00.423439 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.423415 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-483de8-8487964dcd6pnk"] Nov 01 13:17:00.567363 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.567276 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc"] Nov 01 13:17:00.570480 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:00.570457 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-483de8-997dd4v7rc"] Nov 01 13:17:01.736953 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:01.736916 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:17:02.043367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:02.043262 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8"] Nov 01 13:17:02.089506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:02.089474 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5bnb8"] Nov 01 13:17:02.396372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:02.394159 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb3c75e-2383-4536-aa9a-4a0d42c143ec" path="/var/lib/kubelet/pods/4eb3c75e-2383-4536-aa9a-4a0d42c143ec/volumes" Nov 01 13:17:02.396372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:02.394874 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" path="/var/lib/kubelet/pods/63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9/volumes" Nov 01 13:17:02.396372 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:02.395504 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" path="/var/lib/kubelet/pods/b417eb73-fa5b-4be5-888b-3898d766322d/volumes" Nov 01 13:17:04.272522 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:04.272484 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:17:05.108223 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:05.108187 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:17:10.536210 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:10.536175 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:17:12.285468 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:12.285436 4484 scope.go:122] "RemoveContainer" containerID="91ed73c3ff41b1ca79cda17af3fa421cc251cdb163030170bb35cfc2b5ce35d8" Nov 01 13:17:12.294028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:12.294002 4484 scope.go:122] "RemoveContainer" containerID="10c5df29b75c074a3600fef575c07e7adf97b201daa2f6a45385716b03f3b23b" Nov 01 13:17:12.301519 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:12.301500 4484 scope.go:122] "RemoveContainer" containerID="175cb42b41b1ef3bc7f060c9cb8710c78a07868f88c44a650d6d7144747513d5" Nov 01 13:17:12.752277 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:12.752247 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:17:18.639837 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:18.639804 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:17:26.043127 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:26.043086 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg"] Nov 01 13:17:26.044878 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:26.044854 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v7cdg"] Nov 01 13:17:26.367944 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:26.367868 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd72f84d-a962-40d7-805b-ad3344a7e13c" path="/var/lib/kubelet/pods/bd72f84d-a962-40d7-805b-ad3344a7e13c/volumes" Nov 01 13:17:30.251044 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:17:30.251002 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:17:57.884634 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:57.884598 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:17:58.028700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:58.028666 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:17:58.143231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:17:58.143156 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:18:00.252689 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:18:00.252643 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:18:12.341034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:12.341006 4484 scope.go:122] "RemoveContainer" containerID="cf93f525e989a5df161b2d3f1f80b73224cca72fe5f56196acfefd1f10a8399d" Nov 01 13:18:12.349020 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:12.349001 4484 scope.go:122] "RemoveContainer" containerID="04bdb71a03db86f03df268a8e19cd0bf4475fa890c2493370b6cc62e78b5acac" Nov 01 13:18:18.993316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:18.993265 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:18:30.251390 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:18:30.251292 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:18:47.058711 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:47.058676 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:18:49.733431 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:49.733389 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-8lq5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 13:18:49.733839 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:49.733459 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-8lq5p" podUID="83638274-23ff-46a4-bcda-875759a4b92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 13:18:49.777330 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:49.776425 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gkqlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 13:18:49.777330 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:49.776526 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gkqlz" podUID="75f922a6-c170-4d88-a0e5-45c2b4e6b763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 13:18:51.416883 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.416846 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm"] Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417203 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" containerName="payload-processing" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417213 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" containerName="payload-processing" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417228 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417233 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417249 4484 cpu_manager.go:414] "containerMap: removing container" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerName="payload-pre-processing" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417254 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerName="payload-pre-processing" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417268 4484 cpu_manager.go:414] "containerMap: removing container" podUID="479a9400-687c-499c-bb8a-16c129d665a6" containerName="revoke-keys" Nov 01 13:18:51.417280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417273 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479a9400-687c-499c-bb8a-16c129d665a6" containerName="revoke-keys" Nov 01 13:18:51.417633 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417335 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="479a9400-687c-499c-bb8a-16c129d665a6" containerName="revoke-keys" Nov 01 13:18:51.417633 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417348 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b417eb73-fa5b-4be5-888b-3898d766322d" containerName="payload-processing" Nov 01 13:18:51.417633 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417354 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" Nov 01 13:18:51.417633 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417361 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" Nov 01 13:18:51.417633 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.417368 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cdd4be-9ee6-48fe-bcf9-b9e27f90a6a9" containerName="payload-pre-processing" Nov 01 13:18:51.420893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.420870 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.469274 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.469243 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:18:51.505636 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.505605 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.505814 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.505737 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.505814 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.505782 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtbf\" (UniqueName: \"kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.515296 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.515269 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:18:51.515296 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.515283 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:18:51.515296 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.515296 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:18:51.517112 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.517094 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm"] Nov 01 13:18:51.607480 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.607448 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.607681 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.607546 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.607681 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.607579 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtbf\" (UniqueName: \"kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.608264 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.608239 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.609826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.609800 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.661317 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.661271 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtbf\" (UniqueName: \"kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf\") pod \"maas-api-revoke-sub-d576e68b-xb5cm\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:51.733584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:51.733551 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:52.096843 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:52.096811 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm"] Nov 01 13:18:52.104366 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:18:52.099576 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21016c53_2912_4545_a961_06703bc5ef1c.slice/crio-7e2ba1c2bdcbe2fb335964b1ace7a8eacf23810ebe8223de85bbf52cdca5da5e WatchSource:0}: Error finding container 7e2ba1c2bdcbe2fb335964b1ace7a8eacf23810ebe8223de85bbf52cdca5da5e: Status 404 returned error can't find the container with id 7e2ba1c2bdcbe2fb335964b1ace7a8eacf23810ebe8223de85bbf52cdca5da5e Nov 01 13:18:52.828638 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:52.828603 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" event={"ID":"21016c53-2912-4545-a961-06703bc5ef1c","Type":"ContainerStarted","Data":"ad7036aee4bde10b72432a1ff15b15809d85d992762896b960744d4065dd9e2e"} Nov 01 13:18:52.829166 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:52.828663 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" event={"ID":"21016c53-2912-4545-a961-06703bc5ef1c","Type":"ContainerStarted","Data":"7e2ba1c2bdcbe2fb335964b1ace7a8eacf23810ebe8223de85bbf52cdca5da5e"} Nov 01 13:18:52.935556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:52.935506 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" podStartSLOduration=1.9354929539999999 podStartE2EDuration="1.935492954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:18:52.933400503 +0000 UTC m=+3402.683343481" watchObservedRunningTime="2026-11-01 13:18:52.935492954 +0000 UTC m=+3402.685435931" Nov 01 13:18:53.834468 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:53.834432 4484 generic.go:356] "Generic (PLEG): container finished" podID="21016c53-2912-4545-a961-06703bc5ef1c" containerID="ad7036aee4bde10b72432a1ff15b15809d85d992762896b960744d4065dd9e2e" exitCode=0 Nov 01 13:18:53.834838 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:53.834491 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" event={"ID":"21016c53-2912-4545-a961-06703bc5ef1c","Type":"ContainerDied","Data":"ad7036aee4bde10b72432a1ff15b15809d85d992762896b960744d4065dd9e2e"} Nov 01 13:18:54.974996 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:54.974969 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:55.138997 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.138920 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf\" (UniqueName: \"kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf\") pod \"21016c53-2912-4545-a961-06703bc5ef1c\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " Nov 01 13:18:55.139153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.139042 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca\") pod \"21016c53-2912-4545-a961-06703bc5ef1c\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " Nov 01 13:18:55.139153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.139082 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca\") pod \"21016c53-2912-4545-a961-06703bc5ef1c\" (UID: \"21016c53-2912-4545-a961-06703bc5ef1c\") " Nov 01 13:18:55.139507 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.139465 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca" pod "21016c53-2912-4545-a961-06703bc5ef1c" (UID: "21016c53-2912-4545-a961-06703bc5ef1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:18:55.140986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.140961 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf" pod "21016c53-2912-4545-a961-06703bc5ef1c" (UID: "21016c53-2912-4545-a961-06703bc5ef1c"). InnerVolumeSpecName "kube-api-access-pwtbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:18:55.141091 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.141037 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca" pod "21016c53-2912-4545-a961-06703bc5ef1c" (UID: "21016c53-2912-4545-a961-06703bc5ef1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:18:55.239783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.239739 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtbf\" (UniqueName: \"kubernetes.io/projected/21016c53-2912-4545-a961-06703bc5ef1c-kube-api-access-pwtbf\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:18:55.239783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.239775 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:18:55.239783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.239789 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21016c53-2912-4545-a961-06703bc5ef1c-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:18:55.852270 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.852243 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" Nov 01 13:18:55.852432 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.852240 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm" event={"ID":"21016c53-2912-4545-a961-06703bc5ef1c","Type":"ContainerDied","Data":"7e2ba1c2bdcbe2fb335964b1ace7a8eacf23810ebe8223de85bbf52cdca5da5e"} Nov 01 13:18:55.852432 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:55.852356 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2ba1c2bdcbe2fb335964b1ace7a8eacf23810ebe8223de85bbf52cdca5da5e" Nov 01 13:18:58.042338 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:58.042285 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk"] Nov 01 13:18:58.088442 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:58.088411 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8rhqk"] Nov 01 13:18:58.367680 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:18:58.367605 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a584c9eb-e3a3-4438-8540-4829517114f3" path="/var/lib/kubelet/pods/a584c9eb-e3a3-4438-8540-4829517114f3/volumes" Nov 01 13:19:00.250789 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:19:00.250749 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:19:04.231745 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:04.231708 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:19:09.043796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:09.043761 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r"] Nov 01 13:19:09.089828 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:09.089801 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wvg9r"] Nov 01 13:19:10.388778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:10.388398 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853fa2e7-bced-4b32-ab3c-dd8a4208f666" path="/var/lib/kubelet/pods/853fa2e7-bced-4b32-ab3c-dd8a4208f666/volumes" Nov 01 13:19:12.370316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:12.370272 4484 scope.go:122] "RemoveContainer" containerID="944be714a255b5b8873e2452e10c946dba92d01a5e14e46cb43ba440b163f2ef" Nov 01 13:19:12.378475 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:12.378443 4484 scope.go:122] "RemoveContainer" containerID="d1a3a2cf2b4d5dafaf4bb76ffaef725037e8332eb1f42341cc1a2c22191f6340" Nov 01 13:19:12.385823 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:12.385802 4484 scope.go:122] "RemoveContainer" containerID="eaeedcd348c0cf34319e18972811b0311e15b2c4999b1f1dee7c240da693d07b" Nov 01 13:19:13.943055 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:13.943026 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:19:19.319887 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:19.319840 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:20.307787 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.307753 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6"] Nov 01 13:19:20.308158 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.308146 4484 cpu_manager.go:414] "containerMap: removing container" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" Nov 01 13:19:20.308213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.308159 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafdc18f-06c8-4ad2-8041-f3dc1d1e8d91" containerName="istio-proxy" Nov 01 13:19:20.308213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.308171 4484 cpu_manager.go:414] "containerMap: removing container" podUID="21016c53-2912-4545-a961-06703bc5ef1c" containerName="revoke-keys" Nov 01 13:19:20.308213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.308176 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21016c53-2912-4545-a961-06703bc5ef1c" containerName="revoke-keys" Nov 01 13:19:20.308326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.308248 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="21016c53-2912-4545-a961-06703bc5ef1c" containerName="revoke-keys" Nov 01 13:19:20.311161 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.311138 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.360063 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.360039 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:19:20.360361 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.360074 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:19:20.392052 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.392025 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.392155 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.392072 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.392346 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.392321 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw465\" (UniqueName: \"kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.406381 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.406358 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:19:20.406459 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.406359 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:19:20.408327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.408288 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6"] Nov 01 13:19:20.493643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.493616 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.493773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.493649 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.493773 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.493704 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw465\" (UniqueName: \"kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.494327 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.494280 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.496081 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.496062 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.546257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.546238 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw465\" (UniqueName: \"kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465\") pod \"maas-api-revoke-sub-d2bc93bc-fr6t6\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.624271 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.624221 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:20.991551 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:20.991521 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6"] Nov 01 13:19:20.993830 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:19:20.993801 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511aa6f8_d375_46df_aa93_2f9063290327.slice/crio-5ddd3f307fcd2b51949dfffc17b9b0e8df5a2fd266975ef2ad0fac865fdf3dd4 WatchSource:0}: Error finding container 5ddd3f307fcd2b51949dfffc17b9b0e8df5a2fd266975ef2ad0fac865fdf3dd4: Status 404 returned error can't find the container with id 5ddd3f307fcd2b51949dfffc17b9b0e8df5a2fd266975ef2ad0fac865fdf3dd4 Nov 01 13:19:21.973766 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:21.973673 4484 generic.go:356] "Generic (PLEG): container finished" podID="511aa6f8-d375-46df-aa93-2f9063290327" containerID="830ed85bfd5d8281862b001eac41c058d997817434854cc0f75bf4fcd8e004f5" exitCode=0 Nov 01 13:19:21.974166 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:21.973761 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" event={"ID":"511aa6f8-d375-46df-aa93-2f9063290327","Type":"ContainerDied","Data":"830ed85bfd5d8281862b001eac41c058d997817434854cc0f75bf4fcd8e004f5"} Nov 01 13:19:21.974166 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:21.973813 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" event={"ID":"511aa6f8-d375-46df-aa93-2f9063290327","Type":"ContainerStarted","Data":"5ddd3f307fcd2b51949dfffc17b9b0e8df5a2fd266975ef2ad0fac865fdf3dd4"} Nov 01 13:19:23.104630 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.104606 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:23.219247 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.219221 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465\" (UniqueName: \"kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465\") pod \"511aa6f8-d375-46df-aa93-2f9063290327\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " Nov 01 13:19:23.219429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.219341 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca\") pod \"511aa6f8-d375-46df-aa93-2f9063290327\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " Nov 01 13:19:23.219429 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.219383 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca\") pod \"511aa6f8-d375-46df-aa93-2f9063290327\" (UID: \"511aa6f8-d375-46df-aa93-2f9063290327\") " Nov 01 13:19:23.219777 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.219753 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca" pod "511aa6f8-d375-46df-aa93-2f9063290327" (UID: "511aa6f8-d375-46df-aa93-2f9063290327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:19:23.221357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.221331 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca" pod "511aa6f8-d375-46df-aa93-2f9063290327" (UID: "511aa6f8-d375-46df-aa93-2f9063290327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:19:23.221477 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.221460 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465" pod "511aa6f8-d375-46df-aa93-2f9063290327" (UID: "511aa6f8-d375-46df-aa93-2f9063290327"). InnerVolumeSpecName "kube-api-access-kw465". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:19:23.320060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.319999 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:19:23.320060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.320020 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511aa6f8-d375-46df-aa93-2f9063290327-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:19:23.320060 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.320030 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw465\" (UniqueName: \"kubernetes.io/projected/511aa6f8-d375-46df-aa93-2f9063290327-kube-api-access-kw465\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:19:23.983975 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.983935 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" event={"ID":"511aa6f8-d375-46df-aa93-2f9063290327","Type":"ContainerDied","Data":"5ddd3f307fcd2b51949dfffc17b9b0e8df5a2fd266975ef2ad0fac865fdf3dd4"} Nov 01 13:19:23.984146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.983983 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ddd3f307fcd2b51949dfffc17b9b0e8df5a2fd266975ef2ad0fac865fdf3dd4" Nov 01 13:19:23.984146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:23.983944 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6" Nov 01 13:19:30.250899 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:19:30.250859 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:19:32.581096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:32.581061 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:33.080720 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.080684 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j"] Nov 01 13:19:33.081792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.081766 4484 cpu_manager.go:414] "containerMap: removing container" podUID="511aa6f8-d375-46df-aa93-2f9063290327" containerName="revoke-keys" Nov 01 13:19:33.081969 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.081956 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511aa6f8-d375-46df-aa93-2f9063290327" containerName="revoke-keys" Nov 01 13:19:33.082270 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.082255 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="511aa6f8-d375-46df-aa93-2f9063290327" containerName="revoke-keys" Nov 01 13:19:33.086418 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.086393 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.128184 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.128134 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:19:33.158469 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.158437 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f38349-openshift-default-dockercfg-g2bzm\"" Nov 01 13:19:33.161986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.161954 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:33.191627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.191588 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j"] Nov 01 13:19:33.208091 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208057 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208293 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208102 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208293 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208218 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7dg8\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208293 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208278 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208354 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208407 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208488 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208712 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208569 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.208712 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.208620 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309195 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309232 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309257 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309294 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309356 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309393 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309525 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309425 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309532 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309585 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7dg8\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309678 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309717 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.309826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.309772 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.310110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.310034 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.310284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.310266 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.311999 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.311981 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.312067 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.312008 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.366627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.366463 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.368598 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.368567 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7dg8\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8\") pod \"e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.399178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.399141 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:33.543295 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.543258 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:33.823023 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.822994 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j"] Nov 01 13:19:33.826169 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:19:33.826134 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149afcc7_7257_4082_8d89_9d95b7182329.slice/crio-265fd76d2d758d78a5fa1f8ed2c4fef4578c1c55c9cd6284d08166405619789c WatchSource:0}: Error finding container 265fd76d2d758d78a5fa1f8ed2c4fef4578c1c55c9cd6284d08166405619789c: Status 404 returned error can't find the container with id 265fd76d2d758d78a5fa1f8ed2c4fef4578c1c55c9cd6284d08166405619789c Nov 01 13:19:33.828395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.828358 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:19:33.828516 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.828429 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:19:33.828516 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:33.828460 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:19:34.031844 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:34.031810 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" event={"ID":"149afcc7-7257-4082-8d89-9d95b7182329","Type":"ContainerStarted","Data":"2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061"} Nov 01 13:19:34.032046 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:34.031861 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" event={"ID":"149afcc7-7257-4082-8d89-9d95b7182329","Type":"ContainerStarted","Data":"265fd76d2d758d78a5fa1f8ed2c4fef4578c1c55c9cd6284d08166405619789c"} Nov 01 13:19:34.141935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:34.141843 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podStartSLOduration=1.141826956 podStartE2EDuration="1.141826956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:19:34.139979107 +0000 UTC m=+3443.889922084" watchObservedRunningTime="2026-11-01 13:19:34.141826956 +0000 UTC m=+3443.891769933" Nov 01 13:19:34.400033 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:34.399956 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:34.405059 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:34.405034 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:35.037009 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:35.036972 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:35.037834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:35.037817 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:19:36.890757 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:36.890726 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:37.061592 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:37.061557 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:37.204041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:37.203953 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:39.956495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:39.956463 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:40.115888 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.115844 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:40.378086 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.378054 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl"] Nov 01 13:19:40.381654 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.381629 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.476231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.475936 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ec7fa5f4-dockercfg-pdq8b\"" Nov 01 13:19:40.479085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.479058 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl"] Nov 01 13:19:40.479228 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.479192 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f38349\"" Nov 01 13:19:40.481775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.481738 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79f4\" (UniqueName: \"kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4\") pod \"payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.481965 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.481944 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.536821 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.536797 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f"] Nov 01 13:19:40.540330 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.540294 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.583287 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.583265 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79f4\" (UniqueName: \"kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4\") pod \"payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.583415 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.583368 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.583961 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.583944 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.598428 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.598410 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f"] Nov 01 13:19:40.636894 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.636842 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79f4\" (UniqueName: \"kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4\") pod \"payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.684621 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.684592 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.684734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.684624 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zgnp\" (UniqueName: \"kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp\") pod \"payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.694486 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.694468 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:40.786079 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.786036 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.786239 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.786092 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zgnp\" (UniqueName: \"kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp\") pod \"payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.786833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.786812 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.839548 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.839518 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zgnp\" (UniqueName: \"kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp\") pod \"payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:40.853291 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:40.853263 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:41.056689 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:41.056515 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl"] Nov 01 13:19:41.059640 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:19:41.059610 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122f6df2_ddde_4dec_ac44_fee3dc12c7cf.slice/crio-28b41a92a6f370ef281af645a7d835ba7c35fc760a182721208adabf926ca45a WatchSource:0}: Error finding container 28b41a92a6f370ef281af645a7d835ba7c35fc760a182721208adabf926ca45a: Status 404 returned error can't find the container with id 28b41a92a6f370ef281af645a7d835ba7c35fc760a182721208adabf926ca45a Nov 01 13:19:41.217645 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:41.217574 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f"] Nov 01 13:19:41.220487 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:19:41.220457 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1b4e11_7096_41ab_9abd_27fc0fd6116b.slice/crio-28466618d73479641515ab8ce37574efdce5b6a4dff78b0bc47e04dad83852aa WatchSource:0}: Error finding container 28466618d73479641515ab8ce37574efdce5b6a4dff78b0bc47e04dad83852aa: Status 404 returned error can't find the container with id 28466618d73479641515ab8ce37574efdce5b6a4dff78b0bc47e04dad83852aa Nov 01 13:19:42.069505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.069465 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" event={"ID":"122f6df2-ddde-4dec-ac44-fee3dc12c7cf","Type":"ContainerStarted","Data":"a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501"} Nov 01 13:19:42.069968 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.069558 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" event={"ID":"122f6df2-ddde-4dec-ac44-fee3dc12c7cf","Type":"ContainerStarted","Data":"28b41a92a6f370ef281af645a7d835ba7c35fc760a182721208adabf926ca45a"} Nov 01 13:19:42.069968 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.069592 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:42.071154 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.071126 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" event={"ID":"4b1b4e11-7096-41ab-9abd-27fc0fd6116b","Type":"ContainerStarted","Data":"9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2"} Nov 01 13:19:42.071282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.071169 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" event={"ID":"4b1b4e11-7096-41ab-9abd-27fc0fd6116b","Type":"ContainerStarted","Data":"28466618d73479641515ab8ce37574efdce5b6a4dff78b0bc47e04dad83852aa"} Nov 01 13:19:42.071282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.071188 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:42.176064 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.176013 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" podStartSLOduration=2.175997482 podStartE2EDuration="2.175997482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:19:42.174241447 +0000 UTC m=+3451.924184441" watchObservedRunningTime="2026-11-01 13:19:42.175997482 +0000 UTC m=+3451.925940461" Nov 01 13:19:42.280074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.280021 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" podStartSLOduration=2.280008178 podStartE2EDuration="2.280008178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:19:42.279496421 +0000 UTC m=+3452.029439400" watchObservedRunningTime="2026-11-01 13:19:42.280008178 +0000 UTC m=+3452.029951155" Nov 01 13:19:42.868113 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:42.868069 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:43.024187 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:43.024151 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:43.150130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:43.150050 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:47.512511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:47.512473 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:50.750039 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:50.750004 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:19:52.006459 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:52.006424 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:19:53.100168 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:53.100141 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:19:53.100616 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:53.100254 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:19:56.613869 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:19:56.613766 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:00.205073 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:00.204994 4484 generic.go:356] "Generic (PLEG): container finished" podID="149afcc7-7257-4082-8d89-9d95b7182329" containerID="2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061" exitCode=0 Nov 01 13:20:00.205506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:00.205065 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" event={"ID":"149afcc7-7257-4082-8d89-9d95b7182329","Type":"ContainerDied","Data":"2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061"} Nov 01 13:20:00.205506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:00.205475 4484 scope.go:122] "RemoveContainer" containerID="2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061" Nov 01 13:20:00.206627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:00.206603 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:20:00.206685 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:00.206669 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:20:00.206726 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:00.206702 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:20:00.259947 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:20:00.259919 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:20:01.212168 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:01.212136 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" event={"ID":"149afcc7-7257-4082-8d89-9d95b7182329","Type":"ContainerStarted","Data":"1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f"} Nov 01 13:20:01.338845 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:01.338811 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:01.399455 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:01.399349 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:20:01.400541 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:01.400505 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 13:20:01.400701 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:01.400584 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 13:20:01.753781 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:01.753718 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:02.332387 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:02.332355 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:02.399903 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:02.399872 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 13:20:02.400111 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:02.399938 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 13:20:02.468028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:02.467992 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:02.605468 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:02.605387 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:03.400106 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:03.400065 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 13:20:03.400552 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:03.400120 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 13:20:03.863332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:03.862794 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:04.399678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:04.399644 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 13:20:04.399870 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:04.399696 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 13:20:05.037184 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:05.037152 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:20:05.167721 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:05.167663 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:05.400315 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:05.400206 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 13:20:05.400315 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:05.400260 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 13:20:05.911316 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:05.911272 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:06.403494 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:06.403466 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:20:06.404341 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:06.404320 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:20:09.950632 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:09.950596 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:20:10.424009 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:10.423960 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:12.409420 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:12.409387 4484 scope.go:122] "RemoveContainer" containerID="06e916d99832e35ea2ddd00085f422e75089eacecaab8b6a1db2827c37646b1f" Nov 01 13:20:12.417678 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:12.417656 4484 scope.go:122] "RemoveContainer" containerID="9b3b7eec5986594a767408ff59f485b904f5460906888988c862ec453f6d37f8" Nov 01 13:20:14.794399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:14.794365 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:19.315821 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:19.315789 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:19.587228 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:19.587148 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:23.695068 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:23.695033 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:28.202095 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:28.202062 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:30.250990 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:20:30.250954 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:20:32.417211 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:32.417178 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:20:32.610053 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:32.610013 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:37.024586 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:37.024553 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:41.416954 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:41.416916 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:45.032877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.032839 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj"] Nov 01 13:20:45.036191 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.036167 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.084694 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.084665 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:20:45.092344 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.092288 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.092515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.092360 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.092561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.092537 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tv2r\" (UniqueName: \"kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.130909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.130863 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:20:45.130909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.130879 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:20:45.131191 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.130904 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:20:45.132963 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.132938 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj"] Nov 01 13:20:45.193812 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.193774 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.193990 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.193867 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tv2r\" (UniqueName: \"kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.193990 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.193895 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.194557 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.194538 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.196078 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.196054 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.246913 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.246886 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tv2r\" (UniqueName: \"kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r\") pod \"maas-api-revoke-sub-c5d6c5d0-xpvcj\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.349466 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.349377 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:45.712661 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.712633 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj"] Nov 01 13:20:45.715117 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:20:45.715085 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80e62a6_7390_4185_9b53_6c1d44f97f20.slice/crio-640c52f56db7d5ed79ca525aa8af5d34382d5cbf86759b95dbb945239b3deff3 WatchSource:0}: Error finding container 640c52f56db7d5ed79ca525aa8af5d34382d5cbf86759b95dbb945239b3deff3: Status 404 returned error can't find the container with id 640c52f56db7d5ed79ca525aa8af5d34382d5cbf86759b95dbb945239b3deff3 Nov 01 13:20:45.902514 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:45.902484 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:46.433615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:46.433517 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" event={"ID":"b80e62a6-7390-4185-9b53-6c1d44f97f20","Type":"ContainerStarted","Data":"d2e0d452f69dce8e569507745498899b64fbe56c91c676ea9ceb141a699697a1"} Nov 01 13:20:46.433615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:46.433570 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" event={"ID":"b80e62a6-7390-4185-9b53-6c1d44f97f20","Type":"ContainerStarted","Data":"640c52f56db7d5ed79ca525aa8af5d34382d5cbf86759b95dbb945239b3deff3"} Nov 01 13:20:46.540051 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:46.539999 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" podStartSLOduration=2.53998442 podStartE2EDuration="2.53998442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:20:46.539148504 +0000 UTC m=+3516.289091479" watchObservedRunningTime="2026-11-01 13:20:46.53998442 +0000 UTC m=+3516.289927429" Nov 01 13:20:47.440182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:47.440145 4484 generic.go:356] "Generic (PLEG): container finished" podID="b80e62a6-7390-4185-9b53-6c1d44f97f20" containerID="d2e0d452f69dce8e569507745498899b64fbe56c91c676ea9ceb141a699697a1" exitCode=0 Nov 01 13:20:47.440618 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:47.440235 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" event={"ID":"b80e62a6-7390-4185-9b53-6c1d44f97f20","Type":"ContainerDied","Data":"d2e0d452f69dce8e569507745498899b64fbe56c91c676ea9ceb141a699697a1"} Nov 01 13:20:48.581071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.581046 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:48.624905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.624875 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca\") pod \"b80e62a6-7390-4185-9b53-6c1d44f97f20\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " Nov 01 13:20:48.625088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.624935 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca\") pod \"b80e62a6-7390-4185-9b53-6c1d44f97f20\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " Nov 01 13:20:48.625088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.624959 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r\" (UniqueName: \"kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r\") pod \"b80e62a6-7390-4185-9b53-6c1d44f97f20\" (UID: \"b80e62a6-7390-4185-9b53-6c1d44f97f20\") " Nov 01 13:20:48.625368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.625345 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca" pod "b80e62a6-7390-4185-9b53-6c1d44f97f20" (UID: "b80e62a6-7390-4185-9b53-6c1d44f97f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:20:48.627127 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.627091 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca" pod "b80e62a6-7390-4185-9b53-6c1d44f97f20" (UID: "b80e62a6-7390-4185-9b53-6c1d44f97f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:20:48.627127 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.627115 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r" pod "b80e62a6-7390-4185-9b53-6c1d44f97f20" (UID: "b80e62a6-7390-4185-9b53-6c1d44f97f20"). InnerVolumeSpecName "kube-api-access-5tv2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:20:48.726483 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.726447 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:20:48.726483 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.726479 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80e62a6-7390-4185-9b53-6c1d44f97f20-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:20:48.726483 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:48.726489 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tv2r\" (UniqueName: \"kubernetes.io/projected/b80e62a6-7390-4185-9b53-6c1d44f97f20-kube-api-access-5tv2r\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:20:49.451726 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:49.451697 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" Nov 01 13:20:49.451906 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:49.451696 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj" event={"ID":"b80e62a6-7390-4185-9b53-6c1d44f97f20","Type":"ContainerDied","Data":"640c52f56db7d5ed79ca525aa8af5d34382d5cbf86759b95dbb945239b3deff3"} Nov 01 13:20:49.451906 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:49.451817 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640c52f56db7d5ed79ca525aa8af5d34382d5cbf86759b95dbb945239b3deff3" Nov 01 13:20:50.394838 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:50.394753 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:54.899623 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:54.899587 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:56.092649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.092619 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:56.210628 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.210599 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:56.566244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.566199 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv"] Nov 01 13:20:56.571534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.571501 4484 cpu_manager.go:414] "containerMap: removing container" podUID="b80e62a6-7390-4185-9b53-6c1d44f97f20" containerName="revoke-keys" Nov 01 13:20:56.571534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.571531 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80e62a6-7390-4185-9b53-6c1d44f97f20" containerName="revoke-keys" Nov 01 13:20:56.571787 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.571704 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80e62a6-7390-4185-9b53-6c1d44f97f20" containerName="revoke-keys" Nov 01 13:20:56.575229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.575203 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.601355 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.601325 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.601473 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.601359 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.601473 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.601392 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kzz\" (UniqueName: \"kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.670515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.670489 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:20:56.670515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.670490 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:20:56.672617 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.672593 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv"] Nov 01 13:20:56.702352 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.702325 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85kzz\" (UniqueName: \"kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.702485 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.702426 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.702485 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.702449 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.704700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.704677 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.716626 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.716605 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:20:56.716704 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.716622 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:20:56.723153 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.723132 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.755338 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.755287 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kzz\" (UniqueName: \"kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:56.887148 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:56.887077 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:57.253496 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:57.253442 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv"] Nov 01 13:20:57.255876 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:20:57.255846 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e77b8c_c7cf_4f03_84ab_54f6914e9cbd.slice/crio-8e2d06f903292f0fb36d8c86c9473571497e4693d08e15e0b28f96178b107caa WatchSource:0}: Error finding container 8e2d06f903292f0fb36d8c86c9473571497e4693d08e15e0b28f96178b107caa: Status 404 returned error can't find the container with id 8e2d06f903292f0fb36d8c86c9473571497e4693d08e15e0b28f96178b107caa Nov 01 13:20:57.505033 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:57.504960 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" event={"ID":"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd","Type":"ContainerStarted","Data":"d3e87b4f609ef98bfdb558796362a4dbcf288b0b9b903387fea4df25c703c3a8"} Nov 01 13:20:57.505033 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:57.505016 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" event={"ID":"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd","Type":"ContainerStarted","Data":"8e2d06f903292f0fb36d8c86c9473571497e4693d08e15e0b28f96178b107caa"} Nov 01 13:20:57.611751 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:57.611706 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" podStartSLOduration=1.611693806 podStartE2EDuration="1.611693806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:20:57.609551703 +0000 UTC m=+3527.359494681" watchObservedRunningTime="2026-11-01 13:20:57.611693806 +0000 UTC m=+3527.361636763" Nov 01 13:20:58.053530 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:58.053505 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:20:58.510920 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:58.510886 4484 generic.go:356] "Generic (PLEG): container finished" podID="67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" containerID="d3e87b4f609ef98bfdb558796362a4dbcf288b0b9b903387fea4df25c703c3a8" exitCode=0 Nov 01 13:20:58.511391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:58.510932 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" event={"ID":"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd","Type":"ContainerDied","Data":"d3e87b4f609ef98bfdb558796362a4dbcf288b0b9b903387fea4df25c703c3a8"} Nov 01 13:20:59.225644 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.225611 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:59.340399 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.340368 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:59.555846 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.555809 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:20:59.669950 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.669925 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:20:59.737570 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.737542 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca\") pod \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " Nov 01 13:20:59.737747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.737725 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz\" (UniqueName: \"kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz\") pod \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " Nov 01 13:20:59.737814 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.737773 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca\") pod \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\" (UID: \"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd\") " Nov 01 13:20:59.738152 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.738129 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca" pod "67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" (UID: "67e77b8c-c7cf-4f03-84ab-54f6914e9cbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:20:59.739619 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.739593 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca" pod "67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" (UID: "67e77b8c-c7cf-4f03-84ab-54f6914e9cbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:20:59.739865 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.739849 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz" pod "67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" (UID: "67e77b8c-c7cf-4f03-84ab-54f6914e9cbd"). InnerVolumeSpecName "kube-api-access-85kzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:20:59.838957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.838878 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:20:59.838957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.838909 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85kzz\" (UniqueName: \"kubernetes.io/projected/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-kube-api-access-85kzz\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:20:59.838957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:20:59.838921 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:00.252031 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:21:00.251992 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:21:00.521416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:00.521333 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" event={"ID":"67e77b8c-c7cf-4f03-84ab-54f6914e9cbd","Type":"ContainerDied","Data":"8e2d06f903292f0fb36d8c86c9473571497e4693d08e15e0b28f96178b107caa"} Nov 01 13:21:00.521416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:00.521383 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e2d06f903292f0fb36d8c86c9473571497e4693d08e15e0b28f96178b107caa" Nov 01 13:21:00.521416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:00.521338 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv" Nov 01 13:21:01.160615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:01.160584 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:04.201900 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:04.201863 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:07.931075 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:07.931042 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f"] Nov 01 13:21:08.032006 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:08.031970 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl"] Nov 01 13:21:08.259742 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:08.259700 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:08.372477 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:08.372444 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:08.601091 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:08.600988 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:12.439511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:12.439480 4484 scope.go:122] "RemoveContainer" containerID="dd04aa26b8f2cf8cf7eda6bd5ab160c08ec39bd4e920b72a411326f106fe3a3c" Nov 01 13:21:12.495583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:12.495554 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:12.931437 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:12.931399 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" containerID="cri-o://9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2" gracePeriod=25 Nov 01 13:21:13.032914 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:13.032877 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" containerID="cri-o://a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501" gracePeriod=25 Nov 01 13:21:13.098209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:13.098174 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:21:13.098405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:13.098183 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:21:13.919785 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:13.919750 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:14.243273 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:14.243232 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:14.504488 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:14.504408 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv"] Nov 01 13:21:14.550100 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:14.550068 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f38349-4bmfv"] Nov 01 13:21:14.905688 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:14.905613 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:16.049250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:16.049218 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c"] Nov 01 13:21:16.095239 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:16.095190 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccab7d7c-dvt4c"] Nov 01 13:21:16.367536 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:16.367462 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" path="/var/lib/kubelet/pods/67e77b8c-c7cf-4f03-84ab-54f6914e9cbd/volumes" Nov 01 13:21:16.367922 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:16.367908 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4385c0-e814-460b-b898-ae4ac14632d0" path="/var/lib/kubelet/pods/fc4385c0-e814-460b-b898-ae4ac14632d0/volumes" Nov 01 13:21:17.230499 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:17.230461 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:21:19.969148 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:19.969111 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:20.600549 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:20.600517 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:20.717551 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:20.717520 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:20.947923 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:20.947846 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:20.948414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:20.948393 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j"] Nov 01 13:21:20.948615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:20.948596 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" containerID="cri-o://1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f" gracePeriod=30 Nov 01 13:21:21.406506 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:21.406469 4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 13:21:21.406905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:21.406533 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 13:21:23.097826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:23.097790 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:21:23.097826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:23.097805 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 13:21:26.190778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.190753 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:21:26.205469 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205446 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205556 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205507 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205595 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205551 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205635 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205596 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205685 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205662 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205745 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205692 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205743 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205806 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205747 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:21:26.205925 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205804 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.205925 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205811 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:21:26.205925 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205845 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo\") pod \"149afcc7-7257-4082-8d89-9d95b7182329\" (UID: \"149afcc7-7257-4082-8d89-9d95b7182329\") " Nov 01 13:21:26.206096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.205984 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:21:26.206151 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206097 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:21:26.206151 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206121 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:21:26.206275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206239 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.206275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206256 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.206275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206266 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149afcc7-7257-4082-8d89-9d95b7182329-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.206458 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206279 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.206458 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.206292 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.207754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.207726 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "kube-api-access-c7dg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:21:26.207754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.207740 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:21:26.208092 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.208067 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:21:26.208188 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.208090 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo" pod "149afcc7-7257-4082-8d89-9d95b7182329" (UID: "149afcc7-7257-4082-8d89-9d95b7182329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:21:26.307292 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.307267 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.307292 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.307289 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149afcc7-7257-4082-8d89-9d95b7182329-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.307292 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.307314 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7dg8\" (UniqueName: \"kubernetes.io/projected/149afcc7-7257-4082-8d89-9d95b7182329-kube-api-access-c7dg8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.307521 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.307324 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149afcc7-7257-4082-8d89-9d95b7182329-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:26.643709 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.643675 4484 generic.go:356] "Generic (PLEG): container finished" podID="149afcc7-7257-4082-8d89-9d95b7182329" containerID="1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f" exitCode=0 Nov 01 13:21:26.643919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.643742 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" Nov 01 13:21:26.643919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.643761 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" event={"ID":"149afcc7-7257-4082-8d89-9d95b7182329","Type":"ContainerDied","Data":"1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f"} Nov 01 13:21:26.643919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.643835 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j" event={"ID":"149afcc7-7257-4082-8d89-9d95b7182329","Type":"ContainerDied","Data":"265fd76d2d758d78a5fa1f8ed2c4fef4578c1c55c9cd6284d08166405619789c"} Nov 01 13:21:26.643919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.643857 4484 scope.go:122] "RemoveContainer" containerID="1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f" Nov 01 13:21:26.652080 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.652054 4484 scope.go:122] "RemoveContainer" containerID="2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061" Nov 01 13:21:26.659264 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.659249 4484 scope.go:122] "RemoveContainer" containerID="1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f" Nov 01 13:21:26.659561 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:21:26.659535 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f\": container with ID starting with 1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f not found: ID does not exist" containerID="1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f" Nov 01 13:21:26.659641 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.659573 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f"} err="failed to get container status \"1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f\": rpc error: code = NotFound desc = could not find container \"1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f\": container with ID starting with 1d9635f94de7c57e73bb1de4d97df1083cd5166e4b4e4dc44f65aee72529e34f not found: ID does not exist" Nov 01 13:21:26.659641 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.659597 4484 scope.go:122] "RemoveContainer" containerID="2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061" Nov 01 13:21:26.659828 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:21:26.659813 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061\": container with ID starting with 2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061 not found: ID does not exist" containerID="2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061" Nov 01 13:21:26.659885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.659835 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061"} err="failed to get container status \"2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061\": rpc error: code = NotFound desc = could not find container \"2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061\": container with ID starting with 2bd67add97ca2ea02462f0fdfa9a906ee835620417a41dbbd0748dc5e1f2c061 not found: ID does not exist" Nov 01 13:21:26.796758 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.796718 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j"] Nov 01 13:21:26.842811 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:26.842780 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f38349-openshift-default-5468c99ccc8x82j"] Nov 01 13:21:27.164239 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164207 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk"] Nov 01 13:21:27.164595 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164581 4484 cpu_manager.go:414] "containerMap: removing container" podUID="67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" containerName="revoke-keys" Nov 01 13:21:27.164656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164596 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" containerName="revoke-keys" Nov 01 13:21:27.164656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164610 4484 cpu_manager.go:414] "containerMap: removing container" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" Nov 01 13:21:27.164656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164616 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" Nov 01 13:21:27.164656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164634 4484 cpu_manager.go:414] "containerMap: removing container" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" Nov 01 13:21:27.164656 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164639 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" Nov 01 13:21:27.164820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164694 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" Nov 01 13:21:27.164820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.164708 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e77b8c-c7cf-4f03-84ab-54f6914e9cbd" containerName="revoke-keys" Nov 01 13:21:27.169182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.169159 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.217624 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.217591 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9ab48fd4-kserve-self-signed-certs\"" Nov 01 13:21:27.223971 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.223942 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.224088 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.223987 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.224133 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.224093 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.224133 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.224117 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.224238 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.224224 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.224277 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.224250 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5tg\" (UniqueName: \"kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.226735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.226714 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk"] Nov 01 13:21:27.247820 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.247797 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.327710 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.327785 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.327873 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.327898 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.328021 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.328053 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5tg\" (UniqueName: \"kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.328807 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.329511 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.329040 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.334071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.334046 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.336206 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.336180 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.336699 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.336681 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.386253 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.386222 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5tg\" (UniqueName: \"kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg\") pod \"e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.480745 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.480711 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:27.868034 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:27.867957 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk"] Nov 01 13:21:27.871443 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:21:27.871420 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod091b976d_f29d_4d00_b68f_1b7075a05c62.slice/crio-ab99617dfe909d4fa21ba2c9d957681a4eeb8e013a90ef80b7c86e037b60a594 WatchSource:0}: Error finding container ab99617dfe909d4fa21ba2c9d957681a4eeb8e013a90ef80b7c86e037b60a594: Status 404 returned error can't find the container with id ab99617dfe909d4fa21ba2c9d957681a4eeb8e013a90ef80b7c86e037b60a594 Nov 01 13:21:28.367793 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:28.367761 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149afcc7-7257-4082-8d89-9d95b7182329" path="/var/lib/kubelet/pods/149afcc7-7257-4082-8d89-9d95b7182329/volumes" Nov 01 13:21:28.667236 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:28.667146 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" event={"ID":"091b976d-f29d-4d00-b68f-1b7075a05c62","Type":"ContainerStarted","Data":"2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249"} Nov 01 13:21:28.667236 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:28.667204 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" event={"ID":"091b976d-f29d-4d00-b68f-1b7075a05c62","Type":"ContainerStarted","Data":"ab99617dfe909d4fa21ba2c9d957681a4eeb8e013a90ef80b7c86e037b60a594"} Nov 01 13:21:28.775484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:28.775425 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" podStartSLOduration=1.775408647 podStartE2EDuration="1.775408647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:21:28.773460588 +0000 UTC m=+3558.523403568" watchObservedRunningTime="2026-11-01 13:21:28.775408647 +0000 UTC m=+3558.525351829" Nov 01 13:21:29.481067 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:29.481040 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:30.130229 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:30.130191 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:21:30.250961 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:21:30.250926 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:21:33.100686 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:21:33.099920 4484 logging.go:55] [core] [Channel #7901 SubChannel #7902]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.197:50052", ServerName: "10.133.0.197:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.197:50052: connect: connection refused" Nov 01 13:21:33.100686 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:21:33.100621 4484 logging.go:55] [core] [Channel #7903 SubChannel #7904]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.198:50052", ServerName: "10.133.0.198:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.198:50052: connect: connection refused" Nov 01 13:21:33.408723 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.408696 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:21:33.425332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.425292 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:21:33.526581 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.526553 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp\" (UniqueName: \"kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp\") pod \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " Nov 01 13:21:33.526761 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.526652 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume\") pod \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\" (UID: \"4b1b4e11-7096-41ab-9abd-27fc0fd6116b\") " Nov 01 13:21:33.526761 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.526689 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume\") pod \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " Nov 01 13:21:33.526761 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.526741 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4\" (UniqueName: \"kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4\") pod \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\" (UID: \"122f6df2-ddde-4dec-ac44-fee3dc12c7cf\") " Nov 01 13:21:33.527116 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.527083 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume" pod "4b1b4e11-7096-41ab-9abd-27fc0fd6116b" (UID: "4b1b4e11-7096-41ab-9abd-27fc0fd6116b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:21:33.527233 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.527131 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume" pod "122f6df2-ddde-4dec-ac44-fee3dc12c7cf" (UID: "122f6df2-ddde-4dec-ac44-fee3dc12c7cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:21:33.529411 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.529387 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp" pod "4b1b4e11-7096-41ab-9abd-27fc0fd6116b" (UID: "4b1b4e11-7096-41ab-9abd-27fc0fd6116b"). InnerVolumeSpecName "kube-api-access-2zgnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:21:33.529498 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.529481 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4" pod "122f6df2-ddde-4dec-ac44-fee3dc12c7cf" (UID: "122f6df2-ddde-4dec-ac44-fee3dc12c7cf"). InnerVolumeSpecName "kube-api-access-x79f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:21:33.627645 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.627614 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79f4\" (UniqueName: \"kubernetes.io/projected/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-kube-api-access-x79f4\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:33.627645 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.627645 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zgnp\" (UniqueName: \"kubernetes.io/projected/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-kube-api-access-2zgnp\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:33.627892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.627660 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1b4e11-7096-41ab-9abd-27fc0fd6116b-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:33.627892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.627668 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f6df2-ddde-4dec-ac44-fee3dc12c7cf-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:21:33.724951 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.724918 4484 generic.go:356] "Generic (PLEG): container finished" podID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerID="a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501" exitCode=0 Nov 01 13:21:33.725131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.724965 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" Nov 01 13:21:33.725131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.725006 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" event={"ID":"122f6df2-ddde-4dec-ac44-fee3dc12c7cf","Type":"ContainerDied","Data":"a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501"} Nov 01 13:21:33.725131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.725059 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" event={"ID":"122f6df2-ddde-4dec-ac44-fee3dc12c7cf","Type":"ContainerDied","Data":"28b41a92a6f370ef281af645a7d835ba7c35fc760a182721208adabf926ca45a"} Nov 01 13:21:33.725131 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.725075 4484 scope.go:122] "RemoveContainer" containerID="a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501" Nov 01 13:21:33.726288 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.726263 4484 generic.go:356] "Generic (PLEG): container finished" podID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerID="9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2" exitCode=0 Nov 01 13:21:33.726401 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.726334 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" Nov 01 13:21:33.726401 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.726340 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" event={"ID":"4b1b4e11-7096-41ab-9abd-27fc0fd6116b","Type":"ContainerDied","Data":"9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2"} Nov 01 13:21:33.726498 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.726399 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" event={"ID":"4b1b4e11-7096-41ab-9abd-27fc0fd6116b","Type":"ContainerDied","Data":"28466618d73479641515ab8ce37574efdce5b6a4dff78b0bc47e04dad83852aa"} Nov 01 13:21:33.735169 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.735151 4484 scope.go:122] "RemoveContainer" containerID="a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501" Nov 01 13:21:33.735522 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:21:33.735499 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501\": container with ID starting with a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501 not found: ID does not exist" containerID="a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501" Nov 01 13:21:33.735627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.735527 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501"} err="failed to get container status \"a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501\": rpc error: code = NotFound desc = could not find container \"a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501\": container with ID starting with a90c10d3a74839e844404a4976fef3bf5ed55fad11a031d017e55375f4da3501 not found: ID does not exist" Nov 01 13:21:33.735627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.735551 4484 scope.go:122] "RemoveContainer" containerID="9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2" Nov 01 13:21:33.743602 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.743582 4484 scope.go:122] "RemoveContainer" containerID="9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2" Nov 01 13:21:33.743839 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:21:33.743818 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2\": container with ID starting with 9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2 not found: ID does not exist" containerID="9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2" Nov 01 13:21:33.743905 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.743850 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2"} err="failed to get container status \"9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2\": rpc error: code = NotFound desc = could not find container \"9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2\": container with ID starting with 9aee93169fbd32dd3bcb76841d8b59ea271225d8a8bb3a0b6116c87e5f7675c2 not found: ID does not exist" Nov 01 13:21:33.876486 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.876411 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f"] Nov 01 13:21:33.879718 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.879693 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f"] Nov 01 13:21:33.980919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.980891 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl"] Nov 01 13:21:33.982529 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:33.982506 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl"] Nov 01 13:21:34.099835 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:34.099793 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f38349-7d4886hgpl" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.197:50052\" within 1s: context deadline exceeded" Nov 01 13:21:34.100968 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:34.100936 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f38349-89c9b5757hmn9f" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.198:50052\" within 1s: context deadline exceeded" Nov 01 13:21:34.101270 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:21:34.100996 4484 logging.go:55] [core] [Channel #7903 SubChannel #7904]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.198:50052", ServerName: "10.133.0.198:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.198:50052: operation was canceled" Nov 01 13:21:34.367529 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:34.367496 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" path="/var/lib/kubelet/pods/122f6df2-ddde-4dec-ac44-fee3dc12c7cf/volumes" Nov 01 13:21:34.367877 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:34.367864 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" path="/var/lib/kubelet/pods/4b1b4e11-7096-41ab-9abd-27fc0fd6116b/volumes" Nov 01 13:21:37.481390 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:37.481352 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:37.494016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:37.493986 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:37.766092 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:37.766019 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:21:37.945615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:37.945578 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:38.273635 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:38.273607 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:21:51.043234 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:51.043189 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk"] Nov 01 13:21:51.088857 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:51.088813 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-66cbk"] Nov 01 13:21:52.368129 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:52.368093 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479a9400-687c-499c-bb8a-16c129d665a6" path="/var/lib/kubelet/pods/479a9400-687c-499c-bb8a-16c129d665a6/volumes" Nov 01 13:21:52.868157 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:21:52.868126 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:22:00.250688 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:22:00.250652 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:22:06.903002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.902967 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp"] Nov 01 13:22:06.903393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903344 4484 cpu_manager.go:414] "containerMap: removing container" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" Nov 01 13:22:06.903393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903357 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" Nov 01 13:22:06.903393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903369 4484 cpu_manager.go:414] "containerMap: removing container" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" Nov 01 13:22:06.903393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903374 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" Nov 01 13:22:06.903597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903444 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="149afcc7-7257-4082-8d89-9d95b7182329" containerName="istio-proxy" Nov 01 13:22:06.903597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903457 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="122f6df2-ddde-4dec-ac44-fee3dc12c7cf" containerName="payload-pre-processing" Nov 01 13:22:06.903597 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.903464 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1b4e11-7096-41ab-9abd-27fc0fd6116b" containerName="payload-processing" Nov 01 13:22:06.907865 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.907841 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.936528 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.936503 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.936615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.936533 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.936615 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.936560 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.936733 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.936714 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.936796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.936783 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.936833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.936815 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:06.988380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:06.988352 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:22:07.002115 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.002092 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9ab48fd4-kserve-self-signed-certs\"" Nov 01 13:22:07.003943 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.003923 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp"] Nov 01 13:22:07.038028 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038000 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038049 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038121 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038112 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038200 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038170 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038198 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038258 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038242 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038480 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038460 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038585 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038561 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.038585 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.038581 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.040314 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.040284 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.040583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.040563 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.091209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.091175 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n\") pod \"e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.219541 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.219466 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:07.602857 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.602820 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp"] Nov 01 13:22:07.605899 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:22:07.605870 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3be7252_fd23_42ae_9fd6_69afecb8f310.slice/crio-3a5251ea3a9767c984c572878c315ce44e6b136027c3db4925123049a8418e6b WatchSource:0}: Error finding container 3a5251ea3a9767c984c572878c315ce44e6b136027c3db4925123049a8418e6b: Status 404 returned error can't find the container with id 3a5251ea3a9767c984c572878c315ce44e6b136027c3db4925123049a8418e6b Nov 01 13:22:07.895263 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.895168 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" event={"ID":"f3be7252-fd23-42ae-9fd6-69afecb8f310","Type":"ContainerStarted","Data":"2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8"} Nov 01 13:22:07.895263 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:07.895212 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" event={"ID":"f3be7252-fd23-42ae-9fd6-69afecb8f310","Type":"ContainerStarted","Data":"3a5251ea3a9767c984c572878c315ce44e6b136027c3db4925123049a8418e6b"} Nov 01 13:22:08.002899 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:08.002848 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" podStartSLOduration=2.0028346 podStartE2EDuration="2.0028346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:22:08.00143343 +0000 UTC m=+3597.751376411" watchObservedRunningTime="2026-11-01 13:22:08.0028346 +0000 UTC m=+3597.752777595" Nov 01 13:22:08.220143 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:08.220050 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:11.741513 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:11.741478 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:22:12.458419 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:12.458386 4484 scope.go:122] "RemoveContainer" containerID="8d5ab910e0170f987c59f8b0f472b087a77ff27031be042d5380156c6221c45d" Nov 01 13:22:17.220604 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:17.220564 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:17.233115 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:17.233093 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:17.953577 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:17.953547 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:22:20.249619 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:20.249583 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:22:30.250359 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:22:30.250322 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:22:38.853741 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.853703 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5"] Nov 01 13:22:38.857378 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.857353 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:38.905937 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.905911 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:22:38.941415 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.941391 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:38.941553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.941443 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrnx\" (UniqueName: \"kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:38.941553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.941537 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:38.952002 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.951979 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:22:38.952136 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.952009 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:22:38.952136 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.952061 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:22:38.955166 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:38.955140 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5"] Nov 01 13:22:39.042030 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.041994 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.042181 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.042062 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrnx\" (UniqueName: \"kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.042181 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.042095 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.042727 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.042709 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.044315 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.044284 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.094701 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.094677 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrnx\" (UniqueName: \"kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx\") pod \"maas-api-revoke-sub-452b6d7b-hkhq5\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.169999 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.169942 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:39.532800 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:39.532720 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5"] Nov 01 13:22:39.535578 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:22:39.535535 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64901bb9_d4c1_4dd0_bddb_0bfb9aa77e75.slice/crio-2663b6dfebccf5ecc7f1f7d872d5589273352ac2ad10218e874c93e5552c4f16 WatchSource:0}: Error finding container 2663b6dfebccf5ecc7f1f7d872d5589273352ac2ad10218e874c93e5552c4f16: Status 404 returned error can't find the container with id 2663b6dfebccf5ecc7f1f7d872d5589273352ac2ad10218e874c93e5552c4f16 Nov 01 13:22:40.043783 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:40.043748 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" event={"ID":"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75","Type":"ContainerStarted","Data":"9c1789419f71cea8710ac482bdfd3c47e4469c4fe0bb7d0407f6587e01859443"} Nov 01 13:22:40.044236 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:40.043804 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" event={"ID":"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75","Type":"ContainerStarted","Data":"2663b6dfebccf5ecc7f1f7d872d5589273352ac2ad10218e874c93e5552c4f16"} Nov 01 13:22:40.143359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:40.143277 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:22:40.149574 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:40.149515 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" podStartSLOduration=2.149500604 podStartE2EDuration="2.149500604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:22:40.148585633 +0000 UTC m=+3629.898528622" watchObservedRunningTime="2026-11-01 13:22:40.149500604 +0000 UTC m=+3629.899443586" Nov 01 13:22:41.055367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:41.055287 4484 generic.go:356] "Generic (PLEG): container finished" podID="64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" containerID="9c1789419f71cea8710ac482bdfd3c47e4469c4fe0bb7d0407f6587e01859443" exitCode=0 Nov 01 13:22:41.055893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:41.055391 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" event={"ID":"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75","Type":"ContainerDied","Data":"9c1789419f71cea8710ac482bdfd3c47e4469c4fe0bb7d0407f6587e01859443"} Nov 01 13:22:42.521679 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.521640 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:42.625109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.625079 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx\" (UniqueName: \"kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx\") pod \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " Nov 01 13:22:42.625309 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.625159 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca\") pod \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " Nov 01 13:22:42.625309 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.625205 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca\") pod \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\" (UID: \"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75\") " Nov 01 13:22:42.625545 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.625520 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca" pod "64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" (UID: "64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:22:42.627130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.627100 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx" pod "64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" (UID: "64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75"). InnerVolumeSpecName "kube-api-access-pgrnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:22:42.627232 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.627196 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca" pod "64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" (UID: "64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:22:42.726456 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.726425 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgrnx\" (UniqueName: \"kubernetes.io/projected/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-kube-api-access-pgrnx\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:22:42.726659 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.726462 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:22:42.726659 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.726479 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:22:42.812322 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.798462 4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7cxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 13:22:42.812322 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:42.798537 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7cxs" podUID="1f602e80-772b-439c-a04c-526955a37d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 13:22:43.126036 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:43.125942 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" Nov 01 13:22:43.126036 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:43.125981 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5" event={"ID":"64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75","Type":"ContainerDied","Data":"2663b6dfebccf5ecc7f1f7d872d5589273352ac2ad10218e874c93e5552c4f16"} Nov 01 13:22:43.126036 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:43.126030 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2663b6dfebccf5ecc7f1f7d872d5589273352ac2ad10218e874c93e5552c4f16" Nov 01 13:22:51.551953 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:51.551917 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk"] Nov 01 13:22:51.650605 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:51.650562 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:22:55.561286 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:22:55.561205 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:23:00.250054 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:23:00.250012 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:23:05.244960 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:05.244926 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp"] Nov 01 13:23:05.246808 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:05.246776 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:23:06.571480 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.571425 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" podUID="091b976d-f29d-4d00-b68f-1b7075a05c62" containerName="main" containerID="cri-o://2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249" gracePeriod=45 Nov 01 13:23:06.818087 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.818064 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:23:06.860359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860286 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home\") pod \"091b976d-f29d-4d00-b68f-1b7075a05c62\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " Nov 01 13:23:06.860359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860341 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg\" (UniqueName: \"kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg\") pod \"091b976d-f29d-4d00-b68f-1b7075a05c62\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " Nov 01 13:23:06.860550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860382 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs\" (UniqueName: \"kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs\") pod \"091b976d-f29d-4d00-b68f-1b7075a05c62\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " Nov 01 13:23:06.860550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860419 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir\") pod \"091b976d-f29d-4d00-b68f-1b7075a05c62\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " Nov 01 13:23:06.860550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860447 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm\") pod \"091b976d-f29d-4d00-b68f-1b7075a05c62\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " Nov 01 13:23:06.860550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860498 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache\") pod \"091b976d-f29d-4d00-b68f-1b7075a05c62\" (UID: \"091b976d-f29d-4d00-b68f-1b7075a05c62\") " Nov 01 13:23:06.860779 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860541 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home" pod "091b976d-f29d-4d00-b68f-1b7075a05c62" (UID: "091b976d-f29d-4d00-b68f-1b7075a05c62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:06.860779 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860675 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir" pod "091b976d-f29d-4d00-b68f-1b7075a05c62" (UID: "091b976d-f29d-4d00-b68f-1b7075a05c62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:06.860779 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860752 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache" pod "091b976d-f29d-4d00-b68f-1b7075a05c62" (UID: "091b976d-f29d-4d00-b68f-1b7075a05c62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:06.860915 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860823 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:06.860915 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.860842 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:06.862561 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.862535 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs" pod "091b976d-f29d-4d00-b68f-1b7075a05c62" (UID: "091b976d-f29d-4d00-b68f-1b7075a05c62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:23:06.862667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.862593 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm" pod "091b976d-f29d-4d00-b68f-1b7075a05c62" (UID: "091b976d-f29d-4d00-b68f-1b7075a05c62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:06.862667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.862598 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg" pod "091b976d-f29d-4d00-b68f-1b7075a05c62" (UID: "091b976d-f29d-4d00-b68f-1b7075a05c62"). InnerVolumeSpecName "kube-api-access-zp5tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:23:06.961588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.961549 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091b976d-f29d-4d00-b68f-1b7075a05c62-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:06.961588 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.961587 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:06.961770 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.961601 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091b976d-f29d-4d00-b68f-1b7075a05c62-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:06.961770 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:06.961614 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5tg\" (UniqueName: \"kubernetes.io/projected/091b976d-f29d-4d00-b68f-1b7075a05c62-kube-api-access-zp5tg\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:07.257611 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.257575 4484 generic.go:356] "Generic (PLEG): container finished" podID="091b976d-f29d-4d00-b68f-1b7075a05c62" containerID="2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249" exitCode=0 Nov 01 13:23:07.257778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.257625 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" Nov 01 13:23:07.257778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.257654 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" event={"ID":"091b976d-f29d-4d00-b68f-1b7075a05c62","Type":"ContainerDied","Data":"2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249"} Nov 01 13:23:07.257778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.257694 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" event={"ID":"091b976d-f29d-4d00-b68f-1b7075a05c62","Type":"ContainerDied","Data":"ab99617dfe909d4fa21ba2c9d957681a4eeb8e013a90ef80b7c86e037b60a594"} Nov 01 13:23:07.257778 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.257711 4484 scope.go:122] "RemoveContainer" containerID="2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249" Nov 01 13:23:07.266684 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.266669 4484 scope.go:122] "RemoveContainer" containerID="2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249" Nov 01 13:23:07.266921 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:23:07.266904 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249\": container with ID starting with 2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249 not found: ID does not exist" containerID="2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249" Nov 01 13:23:07.266979 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.266929 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249"} err="failed to get container status \"2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249\": rpc error: code = NotFound desc = could not find container \"2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249\": container with ID starting with 2c1318bbd9b99f52935ce98d1ffa28d7c2535a4b11d0203580716c0e19ec1249 not found: ID does not exist" Nov 01 13:23:07.410472 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.410449 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk"] Nov 01 13:23:07.456109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.456088 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk"] Nov 01 13:23:07.755523 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:07.755483 4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-9ab48fd4-kserve-f89f68c77-v8tlk" podUID="091b976d-f29d-4d00-b68f-1b7075a05c62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 13:23:08.028324 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:08.028225 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:23:08.367772 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:08.367694 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091b976d-f29d-4d00-b68f-1b7075a05c62" path="/var/lib/kubelet/pods/091b976d-f29d-4d00-b68f-1b7075a05c62/volumes" Nov 01 13:23:10.834696 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.834663 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5"] Nov 01 13:23:10.835176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.835032 4484 cpu_manager.go:414] "containerMap: removing container" podUID="64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" containerName="revoke-keys" Nov 01 13:23:10.835176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.835043 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" containerName="revoke-keys" Nov 01 13:23:10.835176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.835054 4484 cpu_manager.go:414] "containerMap: removing container" podUID="091b976d-f29d-4d00-b68f-1b7075a05c62" containerName="main" Nov 01 13:23:10.835176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.835059 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091b976d-f29d-4d00-b68f-1b7075a05c62" containerName="main" Nov 01 13:23:10.835176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.835124 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" containerName="revoke-keys" Nov 01 13:23:10.835176 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.835132 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="091b976d-f29d-4d00-b68f-1b7075a05c62" containerName="main" Nov 01 13:23:10.839484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.839463 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:10.887757 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.887733 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:23:10.932202 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.932172 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5s2g\" (UniqueName: \"kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:10.932364 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.932343 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:10.932436 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.932384 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:10.933935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.933916 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:23:10.934012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.933964 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:23:10.934012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.933962 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:23:10.935935 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:10.935916 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5"] Nov 01 13:23:11.033743 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.033713 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.033882 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.033748 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.033882 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.033822 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5s2g\" (UniqueName: \"kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.034492 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.034471 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.036032 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.036015 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.086720 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.086654 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5s2g\" (UniqueName: \"kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g\") pod \"maas-api-revoke-sub-3e31a506-7tnt5\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.150737 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.150711 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:11.516004 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:11.515975 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5"] Nov 01 13:23:11.518770 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:23:11.518742 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993de7ed_f9a7_429d_a3d3_592a02634fcb.slice/crio-0e8521d81d597043c333c9becc05c642fc41088a60b056cfd6cd632403a09b43 WatchSource:0}: Error finding container 0e8521d81d597043c333c9becc05c642fc41088a60b056cfd6cd632403a09b43: Status 404 returned error can't find the container with id 0e8521d81d597043c333c9becc05c642fc41088a60b056cfd6cd632403a09b43 Nov 01 13:23:12.299174 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:12.299137 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" event={"ID":"993de7ed-f9a7-429d-a3d3-592a02634fcb","Type":"ContainerStarted","Data":"2c38b844450a0cc82c58d1afa7c47f31398e86892b25e8c49342aa3df4ed9b5b"} Nov 01 13:23:12.299702 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:12.299222 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" event={"ID":"993de7ed-f9a7-429d-a3d3-592a02634fcb","Type":"ContainerStarted","Data":"0e8521d81d597043c333c9becc05c642fc41088a60b056cfd6cd632403a09b43"} Nov 01 13:23:12.407899 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:12.407832 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" podStartSLOduration=2.407812592 podStartE2EDuration="2.407812592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:23:12.406678629 +0000 UTC m=+3662.156621609" watchObservedRunningTime="2026-11-01 13:23:12.407812592 +0000 UTC m=+3662.157755571" Nov 01 13:23:12.502244 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:12.502206 4484 scope.go:122] "RemoveContainer" containerID="111c5cfa38477a0531915e5905d1586dedea5c85b98c1b416b552a09f18f268a" Nov 01 13:23:12.512231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:12.512208 4484 scope.go:122] "RemoveContainer" containerID="b17d51e68115716ffca529d30c7e0650be8b84818292adcfe1cb689b7443cf5a" Nov 01 13:23:13.314122 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:13.314086 4484 generic.go:356] "Generic (PLEG): container finished" podID="993de7ed-f9a7-429d-a3d3-592a02634fcb" containerID="2c38b844450a0cc82c58d1afa7c47f31398e86892b25e8c49342aa3df4ed9b5b" exitCode=0 Nov 01 13:23:13.314605 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:13.314229 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" event={"ID":"993de7ed-f9a7-429d-a3d3-592a02634fcb","Type":"ContainerDied","Data":"2c38b844450a0cc82c58d1afa7c47f31398e86892b25e8c49342aa3df4ed9b5b"} Nov 01 13:23:14.607103 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.607076 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:14.708085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.708059 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g\" (UniqueName: \"kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g\") pod \"993de7ed-f9a7-429d-a3d3-592a02634fcb\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " Nov 01 13:23:14.708254 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.708095 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca\") pod \"993de7ed-f9a7-429d-a3d3-592a02634fcb\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " Nov 01 13:23:14.708254 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.708173 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca\") pod \"993de7ed-f9a7-429d-a3d3-592a02634fcb\" (UID: \"993de7ed-f9a7-429d-a3d3-592a02634fcb\") " Nov 01 13:23:14.708584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.708560 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca" pod "993de7ed-f9a7-429d-a3d3-592a02634fcb" (UID: "993de7ed-f9a7-429d-a3d3-592a02634fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:23:14.710245 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.710222 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g" pod "993de7ed-f9a7-429d-a3d3-592a02634fcb" (UID: "993de7ed-f9a7-429d-a3d3-592a02634fcb"). InnerVolumeSpecName "kube-api-access-q5s2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:23:14.710352 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.710224 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca" pod "993de7ed-f9a7-429d-a3d3-592a02634fcb" (UID: "993de7ed-f9a7-429d-a3d3-592a02634fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:23:14.808834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.808804 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5s2g\" (UniqueName: \"kubernetes.io/projected/993de7ed-f9a7-429d-a3d3-592a02634fcb-kube-api-access-q5s2g\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:14.808834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.808830 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:14.808992 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:14.808842 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993de7ed-f9a7-429d-a3d3-592a02634fcb-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:15.327584 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:15.327542 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" event={"ID":"993de7ed-f9a7-429d-a3d3-592a02634fcb","Type":"ContainerDied","Data":"0e8521d81d597043c333c9becc05c642fc41088a60b056cfd6cd632403a09b43"} Nov 01 13:23:15.327757 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:15.327599 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e8521d81d597043c333c9becc05c642fc41088a60b056cfd6cd632403a09b43" Nov 01 13:23:15.327757 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:15.327558 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5" Nov 01 13:23:16.653643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:16.653612 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:23:20.264348 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.264294 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" podUID="f3be7252-fd23-42ae-9fd6-69afecb8f310" containerName="main" containerID="cri-o://2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8" gracePeriod=45 Nov 01 13:23:20.523689 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.523633 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:23:20.650178 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.650144 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:23:20.660217 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660192 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n\") pod \"f3be7252-fd23-42ae-9fd6-69afecb8f310\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " Nov 01 13:23:20.660386 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660247 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs\") pod \"f3be7252-fd23-42ae-9fd6-69afecb8f310\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " Nov 01 13:23:20.660386 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660273 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home\") pod \"f3be7252-fd23-42ae-9fd6-69afecb8f310\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " Nov 01 13:23:20.660386 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660293 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm\") pod \"f3be7252-fd23-42ae-9fd6-69afecb8f310\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " Nov 01 13:23:20.660386 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660357 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache\") pod \"f3be7252-fd23-42ae-9fd6-69afecb8f310\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " Nov 01 13:23:20.660637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660397 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir\") pod \"f3be7252-fd23-42ae-9fd6-69afecb8f310\" (UID: \"f3be7252-fd23-42ae-9fd6-69afecb8f310\") " Nov 01 13:23:20.660637 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660589 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home" pod "f3be7252-fd23-42ae-9fd6-69afecb8f310" (UID: "f3be7252-fd23-42ae-9fd6-69afecb8f310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:20.660834 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660815 4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-home\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:20.660908 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660829 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache" pod "f3be7252-fd23-42ae-9fd6-69afecb8f310" (UID: "f3be7252-fd23-42ae-9fd6-69afecb8f310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:20.660908 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.660835 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir" pod "f3be7252-fd23-42ae-9fd6-69afecb8f310" (UID: "f3be7252-fd23-42ae-9fd6-69afecb8f310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:20.663168 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.662680 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs" pod "f3be7252-fd23-42ae-9fd6-69afecb8f310" (UID: "f3be7252-fd23-42ae-9fd6-69afecb8f310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:23:20.663168 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.662791 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm" pod "f3be7252-fd23-42ae-9fd6-69afecb8f310" (UID: "f3be7252-fd23-42ae-9fd6-69afecb8f310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:23:20.663371 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.663350 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n" pod "f3be7252-fd23-42ae-9fd6-69afecb8f310" (UID: "f3be7252-fd23-42ae-9fd6-69afecb8f310"). InnerVolumeSpecName "kube-api-access-xjf5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:23:20.762018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.761992 4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3be7252-fd23-42ae-9fd6-69afecb8f310-tls-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:20.762018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.762018 4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-dshm\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:20.762189 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.762027 4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-model-cache\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:20.762189 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.762036 4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3be7252-fd23-42ae-9fd6-69afecb8f310-tmp-dir\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:20.762189 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:20.762047 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjf5n\" (UniqueName: \"kubernetes.io/projected/f3be7252-fd23-42ae-9fd6-69afecb8f310-kube-api-access-xjf5n\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:21.355515 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.355478 4484 generic.go:356] "Generic (PLEG): container finished" podID="f3be7252-fd23-42ae-9fd6-69afecb8f310" containerID="2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8" exitCode=0 Nov 01 13:23:21.356026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.355525 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" Nov 01 13:23:21.356026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.355564 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" event={"ID":"f3be7252-fd23-42ae-9fd6-69afecb8f310","Type":"ContainerDied","Data":"2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8"} Nov 01 13:23:21.356026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.355620 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp" event={"ID":"f3be7252-fd23-42ae-9fd6-69afecb8f310","Type":"ContainerDied","Data":"3a5251ea3a9767c984c572878c315ce44e6b136027c3db4925123049a8418e6b"} Nov 01 13:23:21.356026 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.355638 4484 scope.go:122] "RemoveContainer" containerID="2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8" Nov 01 13:23:21.365029 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.365011 4484 scope.go:122] "RemoveContainer" containerID="2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8" Nov 01 13:23:21.365327 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:23:21.365289 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8\": container with ID starting with 2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8 not found: ID does not exist" containerID="2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8" Nov 01 13:23:21.365384 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.365336 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8"} err="failed to get container status \"2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8\": rpc error: code = NotFound desc = could not find container \"2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8\": container with ID starting with 2dbe4353cf14dc2de0c6fef4a4fd2ec45d50bdc36d214ab0a58c321c6df1fee8 not found: ID does not exist" Nov 01 13:23:21.507209 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.507175 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp"] Nov 01 13:23:21.553257 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:21.553219 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9ab48fd4-kserve-6cdc48d89c-lbhbp"] Nov 01 13:23:22.366885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:22.366843 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3be7252-fd23-42ae-9fd6-69afecb8f310" path="/var/lib/kubelet/pods/f3be7252-fd23-42ae-9fd6-69afecb8f310/volumes" Nov 01 13:23:25.417268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:25.417236 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:23:30.252122 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:23:30.252084 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:23:35.056247 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:35.056214 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:23:35.386108 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:35.386028 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:23:43.434775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.434745 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn"] Nov 01 13:23:43.435261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.435102 4484 cpu_manager.go:414] "containerMap: removing container" podUID="f3be7252-fd23-42ae-9fd6-69afecb8f310" containerName="main" Nov 01 13:23:43.435261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.435112 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3be7252-fd23-42ae-9fd6-69afecb8f310" containerName="main" Nov 01 13:23:43.435261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.435132 4484 cpu_manager.go:414] "containerMap: removing container" podUID="993de7ed-f9a7-429d-a3d3-592a02634fcb" containerName="revoke-keys" Nov 01 13:23:43.435261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.435138 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993de7ed-f9a7-429d-a3d3-592a02634fcb" containerName="revoke-keys" Nov 01 13:23:43.435261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.435201 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3be7252-fd23-42ae-9fd6-69afecb8f310" containerName="main" Nov 01 13:23:43.435261 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.435211 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="993de7ed-f9a7-429d-a3d3-592a02634fcb" containerName="revoke-keys" Nov 01 13:23:43.439735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.439709 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.509710 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.509674 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.509863 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.509748 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm655\" (UniqueName: \"kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.509912 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.509900 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.534417 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.534390 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:23:43.534417 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.534390 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\"" Nov 01 13:23:43.534587 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.534399 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:23:43.534587 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.534414 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 13:23:43.536424 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.536406 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn"] Nov 01 13:23:43.610957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.610929 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.611085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.610983 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.611085 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.611020 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm655\" (UniqueName: \"kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.611714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.611690 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.613235 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.613213 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.663658 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.663632 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm655\" (UniqueName: \"kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655\") pod \"maas-api-revoke-sub-417878ff-j5mzn\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:43.755510 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:43.755487 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:44.116430 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:44.116407 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn"] Nov 01 13:23:44.119340 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:23:44.119293 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096563af_a9e6_4d0d_92fe_9a343a36bf3d.slice/crio-68fa43c3ac101985775ff99ccc7727cfd962042a342a13a2c12abe57406785b3 WatchSource:0}: Error finding container 68fa43c3ac101985775ff99ccc7727cfd962042a342a13a2c12abe57406785b3: Status 404 returned error can't find the container with id 68fa43c3ac101985775ff99ccc7727cfd962042a342a13a2c12abe57406785b3 Nov 01 13:23:44.476520 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:44.476487 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" event={"ID":"096563af-a9e6-4d0d-92fe-9a343a36bf3d","Type":"ContainerStarted","Data":"93432bc3852eb5b32809b4d81adeeac35702cba746e4c14e9750366a8355509b"} Nov 01 13:23:44.476911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:44.476538 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" event={"ID":"096563af-a9e6-4d0d-92fe-9a343a36bf3d","Type":"ContainerStarted","Data":"68fa43c3ac101985775ff99ccc7727cfd962042a342a13a2c12abe57406785b3"} Nov 01 13:23:44.581720 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:44.581672 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" podStartSLOduration=1.581656598 podStartE2EDuration="1.581656598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:23:44.581552941 +0000 UTC m=+3694.331495919" watchObservedRunningTime="2026-11-01 13:23:44.581656598 +0000 UTC m=+3694.331599576" Nov 01 13:23:45.448048 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:45.448013 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:23:45.482280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:45.482240 4484 generic.go:356] "Generic (PLEG): container finished" podID="096563af-a9e6-4d0d-92fe-9a343a36bf3d" containerID="93432bc3852eb5b32809b4d81adeeac35702cba746e4c14e9750366a8355509b" exitCode=0 Nov 01 13:23:45.482673 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:45.482313 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" event={"ID":"096563af-a9e6-4d0d-92fe-9a343a36bf3d","Type":"ContainerDied","Data":"93432bc3852eb5b32809b4d81adeeac35702cba746e4c14e9750366a8355509b"} Nov 01 13:23:46.612965 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.612942 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:46.738638 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.738570 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655\" (UniqueName: \"kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655\") pod \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " Nov 01 13:23:46.738638 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.738623 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca\") pod \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " Nov 01 13:23:46.738847 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.738693 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca\") pod \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\" (UID: \"096563af-a9e6-4d0d-92fe-9a343a36bf3d\") " Nov 01 13:23:46.739096 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.739070 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca" pod "096563af-a9e6-4d0d-92fe-9a343a36bf3d" (UID: "096563af-a9e6-4d0d-92fe-9a343a36bf3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:23:46.740753 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.740731 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca" pod "096563af-a9e6-4d0d-92fe-9a343a36bf3d" (UID: "096563af-a9e6-4d0d-92fe-9a343a36bf3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:23:46.740895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.740871 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655" pod "096563af-a9e6-4d0d-92fe-9a343a36bf3d" (UID: "096563af-a9e6-4d0d-92fe-9a343a36bf3d"). InnerVolumeSpecName "kube-api-access-pm655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:23:46.840116 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.840091 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm655\" (UniqueName: \"kubernetes.io/projected/096563af-a9e6-4d0d-92fe-9a343a36bf3d-kube-api-access-pm655\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:46.840116 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.840115 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-xks-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:46.840265 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:46.840124 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096563af-a9e6-4d0d-92fe-9a343a36bf3d-maas-api-openshift-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:23:47.492279 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:47.492237 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" event={"ID":"096563af-a9e6-4d0d-92fe-9a343a36bf3d","Type":"ContainerDied","Data":"68fa43c3ac101985775ff99ccc7727cfd962042a342a13a2c12abe57406785b3"} Nov 01 13:23:47.492279 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:47.492282 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68fa43c3ac101985775ff99ccc7727cfd962042a342a13a2c12abe57406785b3" Nov 01 13:23:47.492518 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:47.492247 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn" Nov 01 13:23:55.042943 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:55.042879 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm"] Nov 01 13:23:55.088774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:55.088721 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xb5cm"] Nov 01 13:23:56.207742 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.207700 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:23:56.367077 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.367042 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21016c53-2912-4545-a961-06703bc5ef1c" path="/var/lib/kubelet/pods/21016c53-2912-4545-a961-06703bc5ef1c/volumes" Nov 01 13:23:56.438010 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.437978 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8"] Nov 01 13:23:56.438367 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.438354 4484 cpu_manager.go:414] "containerMap: removing container" podUID="096563af-a9e6-4d0d-92fe-9a343a36bf3d" containerName="revoke-keys" Nov 01 13:23:56.438421 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.438368 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096563af-a9e6-4d0d-92fe-9a343a36bf3d" containerName="revoke-keys" Nov 01 13:23:56.438458 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.438442 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="096563af-a9e6-4d0d-92fe-9a343a36bf3d" containerName="revoke-keys" Nov 01 13:23:56.441699 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.441681 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.458453 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.458403 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6985888875-wxh6v"] Nov 01 13:23:56.461668 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.461651 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.490385 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.490361 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 13:23:56.502700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.502672 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 13:23:56.502799 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.502727 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-758k9\"" Nov 01 13:23:56.523373 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523340 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.523512 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523396 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkj2\" (UniqueName: \"kubernetes.io/projected/39891c8b-f171-4022-977f-89ac5aaad43b-kube-api-access-kqkj2\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.523512 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523491 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39891c8b-f171-4022-977f-89ac5aaad43b-maas-api-tls\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.523660 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523529 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xzm8\" (UniqueName: \"kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.523660 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523636 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.523748 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523672 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39891c8b-f171-4022-977f-89ac5aaad43b-metrics-certs\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.523854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523829 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.523909 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.523867 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39891c8b-f171-4022-977f-89ac5aaad43b-openshift-service-ca\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.536380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.536350 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 13:23:56.536380 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.536356 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 13:23:56.538219 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.538196 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8"] Nov 01 13:23:56.543025 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.543005 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6985888875-wxh6v"] Nov 01 13:23:56.625392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625347 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39891c8b-f171-4022-977f-89ac5aaad43b-maas-api-tls\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.625392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625385 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xzm8\" (UniqueName: \"kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.625669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625420 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.625669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625458 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39891c8b-f171-4022-977f-89ac5aaad43b-metrics-certs\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.625669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625520 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.625669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625546 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39891c8b-f171-4022-977f-89ac5aaad43b-openshift-service-ca\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.625669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625615 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.625669 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.625658 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkj2\" (UniqueName: \"kubernetes.io/projected/39891c8b-f171-4022-977f-89ac5aaad43b-kube-api-access-kqkj2\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.626486 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.626458 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.626700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.626497 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39891c8b-f171-4022-977f-89ac5aaad43b-openshift-service-ca\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.628649 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.628624 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.628814 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.628681 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39891c8b-f171-4022-977f-89ac5aaad43b-maas-api-tls\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.629098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.629079 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.630848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.630822 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39891c8b-f171-4022-977f-89ac5aaad43b-metrics-certs\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.681639 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.681609 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xzm8\" (UniqueName: \"kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8\") pod \"maas-api-6cdfff487-mqrf8\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.681826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.681666 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkj2\" (UniqueName: \"kubernetes.io/projected/39891c8b-f171-4022-977f-89ac5aaad43b-kube-api-access-kqkj2\") pod \"maas-api-6985888875-wxh6v\" (UID: \"39891c8b-f171-4022-977f-89ac5aaad43b\") " pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:56.754357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.754323 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:56.773382 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:56.773362 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:57.148066 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:57.148041 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8"] Nov 01 13:23:57.149972 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:23:57.149944 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfed8db_74ee_48fc_b553_d2c0d350c996.slice/crio-4d3aec58c59a0398c4cecbd51358e273ca109b00709fd2d43013ec5f239d691b WatchSource:0}: Error finding container 4d3aec58c59a0398c4cecbd51358e273ca109b00709fd2d43013ec5f239d691b: Status 404 returned error can't find the container with id 4d3aec58c59a0398c4cecbd51358e273ca109b00709fd2d43013ec5f239d691b Nov 01 13:23:57.151369 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:57.151351 4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 13:23:57.172201 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:57.172177 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6985888875-wxh6v"] Nov 01 13:23:57.186790 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:23:57.186767 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39891c8b_f171_4022_977f_89ac5aaad43b.slice/crio-9980dd39c0bc9eb9c306e3a2f8a05eec56078a0bb6a8dbc7c19b9688e3386c5e WatchSource:0}: Error finding container 9980dd39c0bc9eb9c306e3a2f8a05eec56078a0bb6a8dbc7c19b9688e3386c5e: Status 404 returned error can't find the container with id 9980dd39c0bc9eb9c306e3a2f8a05eec56078a0bb6a8dbc7c19b9688e3386c5e Nov 01 13:23:57.535918 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:57.535881 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" event={"ID":"fdfed8db-74ee-48fc-b553-d2c0d350c996","Type":"ContainerStarted","Data":"4d3aec58c59a0398c4cecbd51358e273ca109b00709fd2d43013ec5f239d691b"} Nov 01 13:23:57.537110 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:57.537082 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" event={"ID":"39891c8b-f171-4022-977f-89ac5aaad43b","Type":"ContainerStarted","Data":"9980dd39c0bc9eb9c306e3a2f8a05eec56078a0bb6a8dbc7c19b9688e3386c5e"} Nov 01 13:23:58.543126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:58.543095 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" event={"ID":"39891c8b-f171-4022-977f-89ac5aaad43b","Type":"ContainerStarted","Data":"2175a00f569c1241408222274d8ca3ee43c7aaeaaf60fbf0ffe92b627783a33c"} Nov 01 13:23:58.543654 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:58.543156 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:23:58.544742 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:58.544718 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" event={"ID":"fdfed8db-74ee-48fc-b553-d2c0d350c996","Type":"ContainerStarted","Data":"42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1"} Nov 01 13:23:58.544885 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:58.544768 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:23:58.651643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:58.651591 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" podStartSLOduration=1.7310853370000001 podStartE2EDuration="2.651576872s" totalImagesPullingTime="920.491535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:23:58.649525952 +0000 UTC m=+3708.399468931" watchObservedRunningTime="2026-11-01 13:23:58.651576872 +0000 UTC m=+3708.401519868" Nov 01 13:23:58.757400 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:23:58.757341 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" podStartSLOduration=1.840057311 podStartE2EDuration="2.75732638s" totalImagesPullingTime="917.269069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:23:58.756028874 +0000 UTC m=+3708.505971847" watchObservedRunningTime="2026-11-01 13:23:58.75732638 +0000 UTC m=+3708.507269354" Nov 01 13:24:00.253591 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:00.253547 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:24:02.863940 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:02.863910 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:03.502749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:03.502715 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:03.675517 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:03.675489 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:04.558430 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:04.558403 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6985888875-wxh6v" Nov 01 13:24:04.558814 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:04.558469 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:24:04.750869 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:04.750839 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8"] Nov 01 13:24:04.751058 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:04.750998 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" podUID="fdfed8db-74ee-48fc-b553-d2c0d350c996" containerName="maas-api" containerID="cri-o://42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1" gracePeriod=30 Nov 01 13:24:05.009289 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.009266 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:24:05.102957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.102932 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls\") pod \"fdfed8db-74ee-48fc-b553-d2c0d350c996\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " Nov 01 13:24:05.103068 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.102974 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca\") pod \"fdfed8db-74ee-48fc-b553-d2c0d350c996\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " Nov 01 13:24:05.103068 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.103053 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs\") pod \"fdfed8db-74ee-48fc-b553-d2c0d350c996\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " Nov 01 13:24:05.103144 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.103114 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8\" (UniqueName: \"kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8\") pod \"fdfed8db-74ee-48fc-b553-d2c0d350c996\" (UID: \"fdfed8db-74ee-48fc-b553-d2c0d350c996\") " Nov 01 13:24:05.103368 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.103340 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca" pod "fdfed8db-74ee-48fc-b553-d2c0d350c996" (UID: "fdfed8db-74ee-48fc-b553-d2c0d350c996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:24:05.103484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.103431 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfed8db-74ee-48fc-b553-d2c0d350c996-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:05.105036 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.105007 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls" pod "fdfed8db-74ee-48fc-b553-d2c0d350c996" (UID: "fdfed8db-74ee-48fc-b553-d2c0d350c996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:24:05.105141 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.105084 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs" pod "fdfed8db-74ee-48fc-b553-d2c0d350c996" (UID: "fdfed8db-74ee-48fc-b553-d2c0d350c996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:24:05.105185 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.105146 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8" pod "fdfed8db-74ee-48fc-b553-d2c0d350c996" (UID: "fdfed8db-74ee-48fc-b553-d2c0d350c996"). InnerVolumeSpecName "kube-api-access-2xzm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:24:05.204016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.203952 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xzm8\" (UniqueName: \"kubernetes.io/projected/fdfed8db-74ee-48fc-b553-d2c0d350c996-kube-api-access-2xzm8\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:05.204016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.203975 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:05.204016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.203986 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdfed8db-74ee-48fc-b553-d2c0d350c996-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:05.614819 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.614786 4484 generic.go:356] "Generic (PLEG): container finished" podID="fdfed8db-74ee-48fc-b553-d2c0d350c996" containerID="42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1" exitCode=0 Nov 01 13:24:05.615205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.614837 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" Nov 01 13:24:05.615205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.614859 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" event={"ID":"fdfed8db-74ee-48fc-b553-d2c0d350c996","Type":"ContainerDied","Data":"42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1"} Nov 01 13:24:05.615205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.614918 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8" event={"ID":"fdfed8db-74ee-48fc-b553-d2c0d350c996","Type":"ContainerDied","Data":"4d3aec58c59a0398c4cecbd51358e273ca109b00709fd2d43013ec5f239d691b"} Nov 01 13:24:05.615205 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.614942 4484 scope.go:122] "RemoveContainer" containerID="42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1" Nov 01 13:24:05.624098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.624083 4484 scope.go:122] "RemoveContainer" containerID="42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1" Nov 01 13:24:05.624398 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:05.624375 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1\": container with ID starting with 42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1 not found: ID does not exist" containerID="42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1" Nov 01 13:24:05.624479 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.624409 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1"} err="failed to get container status \"42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1\": rpc error: code = NotFound desc = could not find container \"42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1\": container with ID starting with 42b5daef1fc3a5615ec3b7443c583862c736587db5da07ce0476cd83809d8ab1 not found: ID does not exist" Nov 01 13:24:05.766893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.766865 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8"] Nov 01 13:24:05.812962 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:05.812935 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-mqrf8"] Nov 01 13:24:06.368113 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:06.368082 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfed8db-74ee-48fc-b553-d2c0d350c996" path="/var/lib/kubelet/pods/fdfed8db-74ee-48fc-b553-d2c0d350c996/volumes" Nov 01 13:24:06.846357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:06.846282 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:07.011462 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:07.011425 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:07.137639 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:07.137560 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:09.864552 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:09.864518 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:10.031313 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.031272 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:10.288986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.288956 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn"] Nov 01 13:24:10.289356 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.289342 4484 cpu_manager.go:414] "containerMap: removing container" podUID="fdfed8db-74ee-48fc-b553-d2c0d350c996" containerName="maas-api" Nov 01 13:24:10.289417 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.289357 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfed8db-74ee-48fc-b553-d2c0d350c996" containerName="maas-api" Nov 01 13:24:10.289455 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.289422 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfed8db-74ee-48fc-b553-d2c0d350c996" containerName="maas-api" Nov 01 13:24:10.293787 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.293767 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.343745 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.343700 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f3c81bef\"" Nov 01 13:24:10.389812 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.389777 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f3c81bef-dockercfg-dwxr7\"" Nov 01 13:24:10.393098 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.393067 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn"] Nov 01 13:24:10.443747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.443720 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8"] Nov 01 13:24:10.449200 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.449173 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.466344 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.466319 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf62z\" (UniqueName: \"kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z\") pod \"payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.466464 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.466371 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.503371 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.503339 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8"] Nov 01 13:24:10.527236 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.527215 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj"] Nov 01 13:24:10.530791 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.530771 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.567324 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.567248 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume\") pod \"payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.567324 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.567295 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf62z\" (UniqueName: \"kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z\") pod \"payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.567467 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.567417 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.567467 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.567447 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljqk\" (UniqueName: \"kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk\") pod \"payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.579025 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.579005 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f3c81bef-serving-cert\"" Nov 01 13:24:10.588166 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.588147 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj"] Nov 01 13:24:10.614682 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.614661 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f3c81bef\"" Nov 01 13:24:10.618602 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.618583 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.620462 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.620441 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf62z\" (UniqueName: \"kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z\") pod \"payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:10.667933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.667904 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljqk\" (UniqueName: \"kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk\") pod \"payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.668074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.667952 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.668128 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.668083 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.668190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.668134 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume\") pod \"payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.668255 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.668203 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.668255 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.668244 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6z5n\" (UniqueName: \"kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.669132 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.669108 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume\") pod \"payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.691440 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.691414 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:10.720759 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.720732 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljqk\" (UniqueName: \"kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk\") pod \"payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.769352 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.770167 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.770237 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.770285 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.770327 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z5n\" (UniqueName: \"kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:10.770746 4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-serving-cert: secret "maas-api-e2e-ait-f3c81bef-serving-cert" not found Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:10.770815 4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls podName:12ff14d9-44d1-47d7-8bf3-203d28a0e37f nodeName:}" failed. No retries permitted until 2026-11-01 13:24:11.270793738 +0000 UTC m=+3721.020736710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls") pod "maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" (UID: "12ff14d9-44d1-47d7-8bf3-203d28a0e37f") : secret "maas-api-e2e-ait-f3c81bef-serving-cert" not found Nov 01 13:24:10.779936 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.771858 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.784321 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.781886 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.826443 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.826372 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6z5n\" (UniqueName: \"kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:10.843707 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.843655 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:10.918507 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:10.910635 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:11.123271 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.123182 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:11.177049 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.177016 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8"] Nov 01 13:24:11.179602 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:24:11.179412 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037dbddd_9393_4094_b07a_124a89045b87.slice/crio-07771306ae8c1a44728a278a075de93fedcf99e3c89c41cf0322ca2b3d3fb619 WatchSource:0}: Error finding container 07771306ae8c1a44728a278a075de93fedcf99e3c89c41cf0322ca2b3d3fb619: Status 404 returned error can't find the container with id 07771306ae8c1a44728a278a075de93fedcf99e3c89c41cf0322ca2b3d3fb619 Nov 01 13:24:11.277603 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.277574 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:11.279684 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.279659 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\") pod \"maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:11.289324 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.289285 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn"] Nov 01 13:24:11.291375 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:24:11.291348 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097f687a_ba48_4faf_ad99_6c64d450f2c7.slice/crio-a6887265d68c230e520d3eaad3d4f26be6c0ff31122fd937658e59c6ca2205ec WatchSource:0}: Error finding container a6887265d68c230e520d3eaad3d4f26be6c0ff31122fd937658e59c6ca2205ec: Status 404 returned error can't find the container with id a6887265d68c230e520d3eaad3d4f26be6c0ff31122fd937658e59c6ca2205ec Nov 01 13:24:11.443084 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.443010 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:11.644673 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.644637 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" event={"ID":"097f687a-ba48-4faf-ad99-6c64d450f2c7","Type":"ContainerStarted","Data":"af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b"} Nov 01 13:24:11.644841 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.644693 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" event={"ID":"097f687a-ba48-4faf-ad99-6c64d450f2c7","Type":"ContainerStarted","Data":"a6887265d68c230e520d3eaad3d4f26be6c0ff31122fd937658e59c6ca2205ec"} Nov 01 13:24:11.644841 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.644719 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:11.646220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.646191 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" event={"ID":"037dbddd-9393-4094-b07a-124a89045b87","Type":"ContainerStarted","Data":"876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba"} Nov 01 13:24:11.646320 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.646249 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" event={"ID":"037dbddd-9393-4094-b07a-124a89045b87","Type":"ContainerStarted","Data":"07771306ae8c1a44728a278a075de93fedcf99e3c89c41cf0322ca2b3d3fb619"} Nov 01 13:24:11.646320 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.646282 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:11.752076 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.752028 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" podStartSLOduration=1.7520154730000002 podStartE2EDuration="1.752015473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:24:11.750499609 +0000 UTC m=+3721.500442588" watchObservedRunningTime="2026-11-01 13:24:11.752015473 +0000 UTC m=+3721.501958450" Nov 01 13:24:11.812641 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.812534 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj"] Nov 01 13:24:11.815034 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:24:11.815011 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ff14d9_44d1_47d7_8bf3_203d28a0e37f.slice/crio-5aa39dde940f0b5b5255b1b244aff01378afb202d87de3ebd84130d686bef64d WatchSource:0}: Error finding container 5aa39dde940f0b5b5255b1b244aff01378afb202d87de3ebd84130d686bef64d: Status 404 returned error can't find the container with id 5aa39dde940f0b5b5255b1b244aff01378afb202d87de3ebd84130d686bef64d Nov 01 13:24:11.857843 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:11.857802 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" podStartSLOduration=1.857789712 podStartE2EDuration="1.857789712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:24:11.855913819 +0000 UTC m=+3721.605856810" watchObservedRunningTime="2026-11-01 13:24:11.857789712 +0000 UTC m=+3721.607732689" Nov 01 13:24:12.652065 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:12.652028 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" event={"ID":"12ff14d9-44d1-47d7-8bf3-203d28a0e37f","Type":"ContainerStarted","Data":"5aa39dde940f0b5b5255b1b244aff01378afb202d87de3ebd84130d686bef64d"} Nov 01 13:24:13.658870 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:13.658837 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" event={"ID":"12ff14d9-44d1-47d7-8bf3-203d28a0e37f","Type":"ContainerStarted","Data":"a5f05d26793fe731a252fabea1fa422c09c5d6525bd3536b2d853294a04d7c6b"} Nov 01 13:24:13.659409 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:13.658995 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:13.783250 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:13.783193 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" podStartSLOduration=2.88733699 podStartE2EDuration="3.783175282s" totalImagesPullingTime="895.838292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:24:13.782889205 +0000 UTC m=+3723.532832182" watchObservedRunningTime="2026-11-01 13:24:13.783175282 +0000 UTC m=+3723.533118259" Nov 01 13:24:15.705553 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:15.705481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:18.818745 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:18.818709 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:24:19.669388 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:19.669361 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:20.210517 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:20.210482 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:20.898334 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:20.898283 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:22.655527 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:22.655494 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:22.656027 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:22.655620 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:24.044220 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:24.044133 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6"] Nov 01 13:24:24.090074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:24.090044 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fr6t6"] Nov 01 13:24:24.368505 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:24.368431 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511aa6f8-d375-46df-aa93-2f9063290327" path="/var/lib/kubelet/pods/511aa6f8-d375-46df-aa93-2f9063290327/volumes" Nov 01 13:24:24.594241 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:24.594209 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:29.456760 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:29.456726 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:30.251582 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:30.251538 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:24:34.099335 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:34.099288 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:38.514720 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:38.514665 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:42.912593 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:42.912551 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:47.407184 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:47.407150 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:49.445456 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:49.445422 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8"] Nov 01 13:24:49.545607 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:49.545577 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn"] Nov 01 13:24:49.766618 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:49.766591 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:49.880343 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:49.880287 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:51.802740 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:51.802709 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:54.445804 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.445761 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" podUID="037dbddd-9393-4094-b07a-124a89045b87" containerName="payload-processing" containerID="cri-o://876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba" gracePeriod=25 Nov 01 13:24:54.546647 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.546610 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" podUID="097f687a-ba48-4faf-ad99-6c64d450f2c7" containerName="payload-pre-processing" containerID="cri-o://af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b" gracePeriod=25 Nov 01 13:24:54.694202 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.694172 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:54.775212 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.775186 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume\") pod \"037dbddd-9393-4094-b07a-124a89045b87\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " Nov 01 13:24:54.775404 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.775345 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk\" (UniqueName: \"kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk\") pod \"037dbddd-9393-4094-b07a-124a89045b87\" (UID: \"037dbddd-9393-4094-b07a-124a89045b87\") " Nov 01 13:24:54.775593 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.775566 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume" pod "037dbddd-9393-4094-b07a-124a89045b87" (UID: "037dbddd-9393-4094-b07a-124a89045b87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:24:54.775714 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.775695 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037dbddd-9393-4094-b07a-124a89045b87-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:54.777275 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.777255 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk" pod "037dbddd-9393-4094-b07a-124a89045b87" (UID: "037dbddd-9393-4094-b07a-124a89045b87"). InnerVolumeSpecName "kube-api-access-sljqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:24:54.792664 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.792645 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:54.855114 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.855077 4484 generic.go:356] "Generic (PLEG): container finished" podID="037dbddd-9393-4094-b07a-124a89045b87" containerID="876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba" exitCode=0 Nov 01 13:24:54.855268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.855127 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" Nov 01 13:24:54.855268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.855159 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" event={"ID":"037dbddd-9393-4094-b07a-124a89045b87","Type":"ContainerDied","Data":"876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba"} Nov 01 13:24:54.855268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.855208 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8" event={"ID":"037dbddd-9393-4094-b07a-124a89045b87","Type":"ContainerDied","Data":"07771306ae8c1a44728a278a075de93fedcf99e3c89c41cf0322ca2b3d3fb619"} Nov 01 13:24:54.855268 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.855232 4484 scope.go:122] "RemoveContainer" containerID="876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba" Nov 01 13:24:54.856673 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.856652 4484 generic.go:356] "Generic (PLEG): container finished" podID="097f687a-ba48-4faf-ad99-6c64d450f2c7" containerID="af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b" exitCode=0 Nov 01 13:24:54.856818 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.856678 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" event={"ID":"097f687a-ba48-4faf-ad99-6c64d450f2c7","Type":"ContainerDied","Data":"af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b"} Nov 01 13:24:54.856941 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.856845 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" event={"ID":"097f687a-ba48-4faf-ad99-6c64d450f2c7","Type":"ContainerDied","Data":"a6887265d68c230e520d3eaad3d4f26be6c0ff31122fd937658e59c6ca2205ec"} Nov 01 13:24:54.856941 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.856713 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn" Nov 01 13:24:54.863830 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.863811 4484 scope.go:122] "RemoveContainer" containerID="876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba" Nov 01 13:24:54.864080 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:54.864061 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba\": container with ID starting with 876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba not found: ID does not exist" containerID="876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba" Nov 01 13:24:54.864126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.864088 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba"} err="failed to get container status \"876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba\": rpc error: code = NotFound desc = could not find container \"876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba\": container with ID starting with 876a0c13de1d18f2f5cb9ec02fcc4e430f0df16bbfe282c0789f918bd49fe4ba not found: ID does not exist" Nov 01 13:24:54.864126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.864103 4484 scope.go:122] "RemoveContainer" containerID="af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b" Nov 01 13:24:54.872215 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.872198 4484 scope.go:122] "RemoveContainer" containerID="af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b" Nov 01 13:24:54.872487 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:24:54.872470 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b\": container with ID starting with af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b not found: ID does not exist" containerID="af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b" Nov 01 13:24:54.872541 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.872492 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b"} err="failed to get container status \"af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b\": rpc error: code = NotFound desc = could not find container \"af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b\": container with ID starting with af99ed93e2b36467dfda928a98962181cdcbfddbae541fd47942ce0a75cb816b not found: ID does not exist" Nov 01 13:24:54.876785 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.876765 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z\" (UniqueName: \"kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z\") pod \"097f687a-ba48-4faf-ad99-6c64d450f2c7\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " Nov 01 13:24:54.876837 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.876807 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume\") pod \"097f687a-ba48-4faf-ad99-6c64d450f2c7\" (UID: \"097f687a-ba48-4faf-ad99-6c64d450f2c7\") " Nov 01 13:24:54.877057 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.877046 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljqk\" (UniqueName: \"kubernetes.io/projected/037dbddd-9393-4094-b07a-124a89045b87-kube-api-access-sljqk\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:54.877190 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.877168 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume" pod "097f687a-ba48-4faf-ad99-6c64d450f2c7" (UID: "097f687a-ba48-4faf-ad99-6c64d450f2c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:24:54.878754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.878738 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z" pod "097f687a-ba48-4faf-ad99-6c64d450f2c7" (UID: "097f687a-ba48-4faf-ad99-6c64d450f2c7"). InnerVolumeSpecName "kube-api-access-pf62z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:24:54.977653 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.977634 4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097f687a-ba48-4faf-ad99-6c64d450f2c7-plugins-config-volume\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:54.977765 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:54.977656 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf62z\" (UniqueName: \"kubernetes.io/projected/097f687a-ba48-4faf-ad99-6c64d450f2c7-kube-api-access-pf62z\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:55.011405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.011378 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8"] Nov 01 13:24:55.015933 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.015901 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f3c81bef-6f6b8f7b75-b7zk8"] Nov 01 13:24:55.271879 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.271806 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn"] Nov 01 13:24:55.276534 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.276509 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f3c81bef-7688f69d7-7qtvn"] Nov 01 13:24:55.603833 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.603767 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj"] Nov 01 13:24:55.604231 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.603967 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" podUID="12ff14d9-44d1-47d7-8bf3-203d28a0e37f" containerName="maas-api" containerID="cri-o://a5f05d26793fe731a252fabea1fa422c09c5d6525bd3536b2d853294a04d7c6b" gracePeriod=30 Nov 01 13:24:55.689957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.689918 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:55.863438 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.863406 4484 generic.go:356] "Generic (PLEG): container finished" podID="12ff14d9-44d1-47d7-8bf3-203d28a0e37f" containerID="a5f05d26793fe731a252fabea1fa422c09c5d6525bd3536b2d853294a04d7c6b" exitCode=0 Nov 01 13:24:55.863550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.863479 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" event={"ID":"12ff14d9-44d1-47d7-8bf3-203d28a0e37f","Type":"ContainerDied","Data":"a5f05d26793fe731a252fabea1fa422c09c5d6525bd3536b2d853294a04d7c6b"} Nov 01 13:24:55.863550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.863533 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" event={"ID":"12ff14d9-44d1-47d7-8bf3-203d28a0e37f","Type":"ContainerDied","Data":"5aa39dde940f0b5b5255b1b244aff01378afb202d87de3ebd84130d686bef64d"} Nov 01 13:24:55.863667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.863556 4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa39dde940f0b5b5255b1b244aff01378afb202d87de3ebd84130d686bef64d" Nov 01 13:24:55.873807 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.873791 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:55.985700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.985662 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs\") pod \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " Nov 01 13:24:55.985892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.985770 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca\") pod \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " Nov 01 13:24:55.985892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.985796 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n\" (UniqueName: \"kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n\") pod \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " Nov 01 13:24:55.985892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.985845 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\") pod \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\" (UID: \"12ff14d9-44d1-47d7-8bf3-203d28a0e37f\") " Nov 01 13:24:55.987226 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.987193 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca" pod "12ff14d9-44d1-47d7-8bf3-203d28a0e37f" (UID: "12ff14d9-44d1-47d7-8bf3-203d28a0e37f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:24:55.989524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.989501 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls" pod "12ff14d9-44d1-47d7-8bf3-203d28a0e37f" (UID: "12ff14d9-44d1-47d7-8bf3-203d28a0e37f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:24:55.989524 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.989511 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n" pod "12ff14d9-44d1-47d7-8bf3-203d28a0e37f" (UID: "12ff14d9-44d1-47d7-8bf3-203d28a0e37f"). InnerVolumeSpecName "kube-api-access-n6z5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:24:55.989854 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:55.989830 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs" pod "12ff14d9-44d1-47d7-8bf3-203d28a0e37f" (UID: "12ff14d9-44d1-47d7-8bf3-203d28a0e37f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 13:24:56.087517 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.087491 4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-openshift-service-ca\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:56.087517 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.087515 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6z5n\" (UniqueName: \"kubernetes.io/projected/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-kube-api-access-n6z5n\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:56.087691 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.087526 4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-maas-api-tls\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:56.087691 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.087536 4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12ff14d9-44d1-47d7-8bf3-203d28a0e37f-metrics-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:24:56.367596 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.367558 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037dbddd-9393-4094-b07a-124a89045b87" path="/var/lib/kubelet/pods/037dbddd-9393-4094-b07a-124a89045b87/volumes" Nov 01 13:24:56.367942 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.367929 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097f687a-ba48-4faf-ad99-6c64d450f2c7" path="/var/lib/kubelet/pods/097f687a-ba48-4faf-ad99-6c64d450f2c7/volumes" Nov 01 13:24:56.868595 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:56.868565 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj" Nov 01 13:24:57.020848 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:57.020817 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj"] Nov 01 13:24:57.066445 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:57.066417 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f3c81bef-7b9fb75b8f-jr4kj"] Nov 01 13:24:57.118484 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:57.118455 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:57.372610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:57.372586 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:24:58.367145 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:58.367113 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ff14d9-44d1-47d7-8bf3-203d28a0e37f" path="/var/lib/kubelet/pods/12ff14d9-44d1-47d7-8bf3-203d28a0e37f/volumes" Nov 01 13:24:58.988273 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:24:58.988244 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:00.159897 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.159785 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:00.250404 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:25:00.250370 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:25:00.654622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.654589 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579"] Nov 01 13:25:00.654957 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.654943 4484 cpu_manager.go:414] "containerMap: removing container" podUID="097f687a-ba48-4faf-ad99-6c64d450f2c7" containerName="payload-pre-processing" Nov 01 13:25:00.655004 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.654958 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097f687a-ba48-4faf-ad99-6c64d450f2c7" containerName="payload-pre-processing" Nov 01 13:25:00.655004 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.654992 4484 cpu_manager.go:414] "containerMap: removing container" podUID="037dbddd-9393-4094-b07a-124a89045b87" containerName="payload-processing" Nov 01 13:25:00.655004 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.654997 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037dbddd-9393-4094-b07a-124a89045b87" containerName="payload-processing" Nov 01 13:25:00.655109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.655016 4484 cpu_manager.go:414] "containerMap: removing container" podUID="12ff14d9-44d1-47d7-8bf3-203d28a0e37f" containerName="maas-api" Nov 01 13:25:00.655109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.655021 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ff14d9-44d1-47d7-8bf3-203d28a0e37f" containerName="maas-api" Nov 01 13:25:00.655109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.655080 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ff14d9-44d1-47d7-8bf3-203d28a0e37f" containerName="maas-api" Nov 01 13:25:00.655109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.655090 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="097f687a-ba48-4faf-ad99-6c64d450f2c7" containerName="payload-pre-processing" Nov 01 13:25:00.655109 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.655097 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="037dbddd-9393-4094-b07a-124a89045b87" containerName="payload-processing" Nov 01 13:25:00.657986 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.657963 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.706775 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.706746 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-624dd4-openshift-default-dockercfg-jzm7v\"" Nov 01 13:25:00.750984 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.750939 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751160 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.750992 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751160 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751090 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751160 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751120 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751412 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751181 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751412 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751225 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751412 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751252 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751412 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751291 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.751648 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.751468 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrcd\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.755765 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.755744 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:00.761349 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.761320 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579"] Nov 01 13:25:00.852482 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852449 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852499 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrcd\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852562 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852629 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852587 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852633 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852657 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852693 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852767 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852738 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.852977 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.852778 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.853172 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.853148 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.853267 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.853189 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.853332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.853277 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.853332 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.853288 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.853612 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.853585 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.854875 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.854852 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.855074 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.855053 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.905409 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.905356 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.906048 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.906023 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrcd\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd\") pod \"e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:00.968424 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.968386 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:00.971789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:00.971502 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:01.387000 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.386978 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579"] Nov 01 13:25:01.389420 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:25:01.389390 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342e684f_1119_4f2e_9f77_0e189842b86c.slice/crio-6bd0b7aa4fb59eef9209ea577974c81e1867733197a535f7991f032f3a87e4c6 WatchSource:0}: Error finding container 6bd0b7aa4fb59eef9209ea577974c81e1867733197a535f7991f032f3a87e4c6: Status 404 returned error can't find the container with id 6bd0b7aa4fb59eef9209ea577974c81e1867733197a535f7991f032f3a87e4c6 Nov 01 13:25:01.391546 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.391512 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:25:01.391651 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.391598 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:25:01.391651 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.391645 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:25:01.895140 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.895104 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" event={"ID":"342e684f-1119-4f2e-9f77-0e189842b86c","Type":"ContainerStarted","Data":"b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9"} Nov 01 13:25:01.895377 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.895157 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" event={"ID":"342e684f-1119-4f2e-9f77-0e189842b86c","Type":"ContainerStarted","Data":"6bd0b7aa4fb59eef9209ea577974c81e1867733197a535f7991f032f3a87e4c6"} Nov 01 13:25:01.972599 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:01.972569 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:02.015016 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:02.014970 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podStartSLOduration=2.014954517 podStartE2EDuration="2.014954517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:25:02.014116276 +0000 UTC m=+3771.764059255" watchObservedRunningTime="2026-11-01 13:25:02.014954517 +0000 UTC m=+3771.764897496" Nov 01 13:25:02.061863 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:02.061836 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:25:02.976987 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:02.976960 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:03.905090 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:03.905064 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:03.905987 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:03.905966 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:04.616940 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:04.616912 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:04.787826 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:04.787796 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:04.921530 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:04.921455 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:07.682734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:07.682703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:07.844280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:07.844229 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:11.242326 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:11.242259 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:11.410872 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:11.410839 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:12.558118 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:12.558085 4484 scope.go:122] "RemoveContainer" containerID="ad7036aee4bde10b72432a1ff15b15809d85d992762896b960744d4065dd9e2e" Nov 01 13:25:15.707040 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:15.706997 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:19.030108 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:19.030074 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:25:20.317896 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:20.317855 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:21.212791 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:21.212741 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:24.822126 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:24.822095 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:28.019383 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:28.019350 4484 generic.go:356] "Generic (PLEG): container finished" podID="342e684f-1119-4f2e-9f77-0e189842b86c" containerID="b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9" exitCode=0 Nov 01 13:25:28.019873 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:28.019410 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" event={"ID":"342e684f-1119-4f2e-9f77-0e189842b86c","Type":"ContainerDied","Data":"b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9"} Nov 01 13:25:28.019873 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:28.019729 4484 scope.go:122] "RemoveContainer" containerID="b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9" Nov 01 13:25:28.020851 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:28.020816 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:25:28.020937 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:28.020891 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:25:28.020937 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:28.020924 4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 13:25:29.026722 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:29.026678 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" event={"ID":"342e684f-1119-4f2e-9f77-0e189842b86c","Type":"ContainerStarted","Data":"a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f"} Nov 01 13:25:29.431780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:29.431703 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:29.559078 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:29.559046 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:29.973023 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:29.972989 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:29.974167 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:29.974145 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 01 13:25:29.974232 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:29.974206 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 01 13:25:30.279789 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:25:30.279750 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:25:30.972754 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:30.972719 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 01 13:25:30.972942 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:30.972773 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 01 13:25:31.972675 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:31.972637 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 01 13:25:31.973082 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:31.972697 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 01 13:25:32.972982 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:32.972944 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 01 13:25:32.973438 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:32.973023 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 01 13:25:33.905713 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:33.905677 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:33.972450 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:33.972416 4484 patch_prober.go:29] interesting pod/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 01 13:25:33.972618 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:33.972470 4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 01 13:25:34.012274 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:34.012234 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:34.976599 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:34.976566 4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:34.977634 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:34.977617 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:25:38.497608 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:38.497552 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:42.910360 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:42.910320 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:47.393749 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:47.393714 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:49.042780 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:49.042745 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj"] Nov 01 13:25:49.088583 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:49.088555 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d6c5d0-xpvcj"] Nov 01 13:25:50.367895 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:50.367865 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80e62a6-7390-4185-9b53-6c1d44f97f20" path="/var/lib/kubelet/pods/b80e62a6-7390-4185-9b53-6c1d44f97f20/volumes" Nov 01 13:25:51.801248 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:51.801211 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:25:56.221114 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:25:56.221030 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:00.255349 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:26:00.255291 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:26:00.605012 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:00.604934 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:05.019606 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:05.019565 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:09.431480 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:09.431447 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:12.586031 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:12.585999 4484 scope.go:122] "RemoveContainer" containerID="830ed85bfd5d8281862b001eac41c058d997817434854cc0f75bf4fcd8e004f5" Nov 01 13:26:13.904519 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:13.904481 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:18.301460 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:18.301431 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:21.555092 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:21.555058 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:22.720391 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:22.720356 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:27.099018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:27.098985 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:30.251457 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:26:30.251418 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:26:31.584283 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:31.584253 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:35.996513 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:35.996476 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:40.503842 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:40.503804 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:44.898959 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:44.898926 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:49.328896 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:49.328856 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:53.849407 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:53.849348 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:58.082157 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:58.082121 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:58.200610 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:58.200577 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:26:58.600853 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:26:58.600817 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:00.251265 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:27:00.251227 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:27:03.014212 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:03.014181 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:04.215345 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:04.215314 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:04.328676 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:04.328637 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:12.014911 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:12.014875 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:12.605464 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:12.605437 4484 scope.go:122] "RemoveContainer" containerID="d3e87b4f609ef98bfdb558796362a4dbcf288b0b9b903387fea4df25c703c3a8" Nov 01 13:27:12.613586 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:12.613563 4484 scope.go:122] "RemoveContainer" containerID="d2e0d452f69dce8e569507745498899b64fbe56c91c676ea9ceb141a699697a1" Nov 01 13:27:13.372868 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:13.372835 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:13.654796 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:13.654721 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:14.371961 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:14.371912 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:17.140336 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:17.140290 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:27:20.347238 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:20.347205 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:20.468533 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:20.468503 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:20.679127 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:20.679054 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579"] Nov 01 13:27:20.679282 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:20.679263 4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" containerID="cri-o://a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f" gracePeriod=30 Nov 01 13:27:20.692682 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:20.692657 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:21.886467 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:21.886437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:27:25.928284 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:25.928258 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:27:26.048406 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048379 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048528 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048426 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048528 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048460 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048528 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048500 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048538 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048567 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048617 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048677 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048669 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048654 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:27:26.048892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048712 4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket\") pod \"342e684f-1119-4f2e-9f77-0e189842b86c\" (UID: \"342e684f-1119-4f2e-9f77-0e189842b86c\") " Nov 01 13:27:26.048892 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048729 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:27:26.049071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048969 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 13:27:26.049071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.048970 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:27:26.049071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.049006 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:27:26.049071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.049034 4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-data\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.049071 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.049070 4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.050789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.050764 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 13:27:26.050894 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.050798 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:27:26.051043 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.051030 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 13:27:26.051146 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.051116 4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd" pod "342e684f-1119-4f2e-9f77-0e189842b86c" (UID: "342e684f-1119-4f2e-9f77-0e189842b86c"). InnerVolumeSpecName "kube-api-access-bgrcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 13:27:26.150000 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.149965 4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-istio-token\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.150000 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.149994 4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgrcd\" (UniqueName: \"kubernetes.io/projected/342e684f-1119-4f2e-9f77-0e189842b86c-kube-api-access-bgrcd\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.150000 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.150005 4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-workload-certs\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.150395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.150014 4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342e684f-1119-4f2e-9f77-0e189842b86c-istio-podinfo\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.150395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.150023 4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-istio-envoy\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.150395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.150031 4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342e684f-1119-4f2e-9f77-0e189842b86c-credential-socket\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.150395 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.150039 4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342e684f-1119-4f2e-9f77-0e189842b86c-istiod-ca-cert\") on node \"ip-10-0-140-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 13:27:26.574520 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.574483 4484 generic.go:356] "Generic (PLEG): container finished" podID="342e684f-1119-4f2e-9f77-0e189842b86c" containerID="a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f" exitCode=0 Nov 01 13:27:26.574735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.574538 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" event={"ID":"342e684f-1119-4f2e-9f77-0e189842b86c","Type":"ContainerDied","Data":"a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f"} Nov 01 13:27:26.574735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.574555 4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" Nov 01 13:27:26.574735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.574588 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579" event={"ID":"342e684f-1119-4f2e-9f77-0e189842b86c","Type":"ContainerDied","Data":"6bd0b7aa4fb59eef9209ea577974c81e1867733197a535f7991f032f3a87e4c6"} Nov 01 13:27:26.574735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.574613 4484 scope.go:122] "RemoveContainer" containerID="a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f" Nov 01 13:27:26.583734 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.583710 4484 scope.go:122] "RemoveContainer" containerID="b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9" Nov 01 13:27:26.593005 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.592976 4484 scope.go:122] "RemoveContainer" containerID="a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f" Nov 01 13:27:26.593295 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:27:26.593273 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f\": container with ID starting with a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f not found: ID does not exist" containerID="a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f" Nov 01 13:27:26.593426 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.593316 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f"} err="failed to get container status \"a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f\": rpc error: code = NotFound desc = could not find container \"a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f\": container with ID starting with a594edc6cc4203915d341c8973abee16d37ed2df3f3c7e9ba74c7b6d17b17b0f not found: ID does not exist" Nov 01 13:27:26.593426 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.593335 4484 scope.go:122] "RemoveContainer" containerID="b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9" Nov 01 13:27:26.593616 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:27:26.593597 4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9\": container with ID starting with b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9 not found: ID does not exist" containerID="b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9" Nov 01 13:27:26.593700 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.593618 4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9"} err="failed to get container status \"b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9\": rpc error: code = NotFound desc = could not find container \"b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9\": container with ID starting with b3e431a33a05d358ccbac37605089b64b3c487a4cf9c2edd2b05be0b4631a7c9 not found: ID does not exist" Nov 01 13:27:26.726565 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.726531 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579"] Nov 01 13:27:26.773542 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:26.773510 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-624dd4-openshift-default-774cf4bb89-jq579"] Nov 01 13:27:28.389336 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:28.389274 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" path="/var/lib/kubelet/pods/342e684f-1119-4f2e-9f77-0e189842b86c/volumes" Nov 01 13:27:30.250322 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:27:30.250264 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:27:31.765737 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:31.765704 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:27:42.654643 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:42.654612 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:27:43.042965 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:43.042930 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5"] Nov 01 13:27:43.089657 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:43.089615 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hkhq5"] Nov 01 13:27:44.379679 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:27:44.379644 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75" path="/var/lib/kubelet/pods/64901bb9-d4c1-4dd0-bddb-0bfb9aa77e75/volumes" Nov 01 13:28:00.250360 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:28:00.250296 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:28:03.630840 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:03.630802 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:28:15.044134 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:15.044098 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5"] Nov 01 13:28:15.089280 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:15.089250 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7tnt5"] Nov 01 13:28:16.368774 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:16.368725 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993de7ed-f9a7-429d-a3d3-592a02634fcb" path="/var/lib/kubelet/pods/993de7ed-f9a7-429d-a3d3-592a02634fcb/volumes" Nov 01 13:28:22.264747 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:22.264712 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:28:30.250624 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:28:30.250585 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:28:34.137739 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:34.137703 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wmb4b"] Nov 01 13:28:43.536400 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:43.536359 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-549c4cbc45-hsm9p_9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b/manager/0.log" Nov 01 13:28:43.874227 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:43.874154 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b5zjm_3f1e9d53-dcee-415e-83bb-bd112a38ee67/manager/0.log" Nov 01 13:28:44.631056 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:44.631029 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57458d666c-fg9qf_4584db68-8ecc-4eff-9658-108b668d04d6/manager/0.log" Nov 01 13:28:46.481405 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:46.481374 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6985888875-wxh6v_39891c8b-f171-4022-977f-89ac5aaad43b/maas-api/0.log" Nov 01 13:28:46.912354 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:46.912255 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-417878ff-j5mzn_096563af-a9e6-4d0d-92fe-9a343a36bf3d/revoke-keys/0.log" Nov 01 13:28:47.043199 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:47.043169 4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn"] Nov 01 13:28:47.088723 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:47.088693 4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417878ff-j5mzn"] Nov 01 13:28:48.367356 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:48.367327 4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096563af-a9e6-4d0d-92fe-9a343a36bf3d" path="/var/lib/kubelet/pods/096563af-a9e6-4d0d-92fe-9a343a36bf3d/volumes" Nov 01 13:28:50.315690 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:50.315653 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wmb4b_3a80816c-07b3-4b1a-9cf5-120d71f4a79b/limitador/0.log" Nov 01 13:28:51.523893 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:51.523861 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-97k8q_f35ed7ef-9b70-4a47-85f6-c5496a308f2a/discovery/0.log" Nov 01 13:28:51.942546 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:51.942475 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf7dc98fb-t2q7w_75a64131-4c62-4bdf-abb6-9c837f6dceba/kube-auth-proxy/0.log" Nov 01 13:28:52.373795 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:52.373765 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-8lq5p_83638274-23ff-46a4-bcda-875759a4b92a/istio-proxy/0.log" Nov 01 13:28:53.034792 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:53.034757 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-f7cxs_1f602e80-772b-439c-a04c-526955a37d83/istio-proxy/0.log" Nov 01 13:28:53.450751 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:53.450671 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gkqlz_75f922a6-c170-4d88-a0e5-45c2b4e6b763/istio-proxy/0.log" Nov 01 13:28:54.511560 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:54.511532 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qbnms_4c72fad4-669b-426c-8c56-153c26c4608b/payload-pre-processing/0.log" Nov 01 13:28:54.712676 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:54.712631 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ph5fh_2478460e-643b-4c88-ba89-bfa8fbc9a2ae/payload-processing/0.log" Nov 01 13:28:55.905235 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:55.905204 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-wpvz4_5454642c-8004-4031-ae58-5f6910da0599/main/0.log" Nov 01 13:28:56.337185 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:56.337156 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7jk9p_28c4ca14-c2bc-4d22-82b9-0adc9acadf02/main/0.log" Nov 01 13:28:56.755018 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:56.754980 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mrlsf_b4fa5ee6-c7c7-419d-9370-5048be88c10d/main/0.log" Nov 01 13:28:57.177062 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:28:57.176981 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh_b95d7585-b83b-4e4a-bfac-cf3c16c721d3/main/0.log" Nov 01 13:29:00.251049 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:29:00.251014 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:29:06.707094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:06.707066 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-549c4cbc45-hsm9p_9c9d8c16-2bd3-4723-8897-d4fa8c2cb88b/manager/0.log" Nov 01 13:29:07.056691 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:07.056663 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b5zjm_3f1e9d53-dcee-415e-83bb-bd112a38ee67/manager/0.log" Nov 01 13:29:07.795359 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:07.795328 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57458d666c-fg9qf_4584db68-8ecc-4eff-9658-108b668d04d6/manager/0.log" Nov 01 13:29:09.255503 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:09.255472 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6985888875-wxh6v_39891c8b-f171-4022-977f-89ac5aaad43b/maas-api/0.log" Nov 01 13:29:12.444108 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:12.444075 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wmb4b_3a80816c-07b3-4b1a-9cf5-120d71f4a79b/limitador/0.log" Nov 01 13:29:12.644752 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:12.644724 4484 scope.go:122] "RemoveContainer" containerID="2c38b844450a0cc82c58d1afa7c47f31398e86892b25e8c49342aa3df4ed9b5b" Nov 01 13:29:12.653075 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:12.653058 4484 scope.go:122] "RemoveContainer" containerID="9c1789419f71cea8710ac482bdfd3c47e4469c4fe0bb7d0407f6587e01859443" Nov 01 13:29:13.479084 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:13.479057 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-97k8q_f35ed7ef-9b70-4a47-85f6-c5496a308f2a/discovery/0.log" Nov 01 13:29:13.902573 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:13.902500 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf7dc98fb-t2q7w_75a64131-4c62-4bdf-abb6-9c837f6dceba/kube-auth-proxy/0.log" Nov 01 13:29:13.996347 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:13.996318 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:29:14.337393 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:14.337366 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-8lq5p_83638274-23ff-46a4-bcda-875759a4b92a/istio-proxy/0.log" Nov 01 13:29:14.970644 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:14.970617 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-f7cxs_1f602e80-772b-439c-a04c-526955a37d83/istio-proxy/0.log" Nov 01 13:29:15.387149 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:15.387121 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gkqlz_75f922a6-c170-4d88-a0e5-45c2b4e6b763/istio-proxy/0.log" Nov 01 13:29:16.463855 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:16.463826 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qbnms_4c72fad4-669b-426c-8c56-153c26c4608b/payload-pre-processing/0.log" Nov 01 13:29:16.660665 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:16.660624 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ph5fh_2478460e-643b-4c88-ba89-bfa8fbc9a2ae/payload-processing/0.log" Nov 01 13:29:17.535715 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:17.535668 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:29:17.582130 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:17.582105 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-wpvz4_5454642c-8004-4031-ae58-5f6910da0599/main/0.log" Nov 01 13:29:17.647240 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:17.647199 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:29:17.995104 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:17.995074 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7jk9p_28c4ca14-c2bc-4d22-82b9-0adc9acadf02/main/0.log" Nov 01 13:29:18.406735 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:18.406655 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mrlsf_b4fa5ee6-c7c7-419d-9370-5048be88c10d/main/0.log" Nov 01 13:29:18.845022 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:18.844991 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8lxkh_b95d7585-b83b-4e4a-bfac-cf3c16c721d3/main/0.log" Nov 01 13:29:22.548423 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:22.548396 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:29:30.254167 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:29:30.254130 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:29:45.198635 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:45.198602 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57458d666c-fg9qf_4584db68-8ecc-4eff-9658-108b668d04d6/manager/0.log" Nov 01 13:29:45.728624 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:45.728599 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57458d666c-fg9qf_4584db68-8ecc-4eff-9658-108b668d04d6/manager/0.log" Nov 01 13:29:46.444118 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:46.444091 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ph5fh_2478460e-643b-4c88-ba89-bfa8fbc9a2ae/payload-processing/0.log" Nov 01 13:29:46.663230 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:29:46.663202 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ph5fh_2478460e-643b-4c88-ba89-bfa8fbc9a2ae/payload-processing/0.log" Nov 01 13:30:00.251884 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:30:00.251849 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:30:12.674667 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:12.674637 4484 scope.go:122] "RemoveContainer" containerID="93432bc3852eb5b32809b4d81adeeac35702cba746e4c14e9750366a8355509b" Nov 01 13:30:20.261414 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:20.261383 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57458d666c-fg9qf_4584db68-8ecc-4eff-9658-108b668d04d6/manager/0.log" Nov 01 13:30:20.794550 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:20.794520 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57458d666c-fg9qf_4584db68-8ecc-4eff-9658-108b668d04d6/manager/0.log" Nov 01 13:30:21.493119 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:21.493090 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ph5fh_2478460e-643b-4c88-ba89-bfa8fbc9a2ae/payload-processing/0.log" Nov 01 13:30:21.702454 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:21.702427 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ph5fh_2478460e-643b-4c88-ba89-bfa8fbc9a2ae/payload-processing/0.log" Nov 01 13:30:22.060218 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:22.060187 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:30:22.189799 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:22.189767 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:30:22.865492 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:22.865437 4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" limit=1024 Nov 01 13:30:30.250763 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:30:30.250721 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor" Nov 01 13:30:34.405789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:34.405758 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x8prg_f93d1106-ec10-4183-b91c-bdbf0ea5fa9a/global-pull-secret-syncer/0.log" Nov 01 13:30:36.244182 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:36.244149 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6kq7t_a9170b98-0937-44e0-86d9-73821df95fef/konnectivity-agent/0.log" Nov 01 13:30:38.285357 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:38.285328 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-174.us-west-2.compute.internal_2c12124bf0630fd70bf94c8ced58f02e/haproxy/0.log" Nov 01 13:30:43.336240 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336200 4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29"] Nov 01 13:30:43.336789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336757 4484 cpu_manager.go:414] "containerMap: removing container" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" Nov 01 13:30:43.336789 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336776 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" Nov 01 13:30:43.336919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336797 4484 cpu_manager.go:414] "containerMap: removing container" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" Nov 01 13:30:43.336919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336805 4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" Nov 01 13:30:43.336919 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336913 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" Nov 01 13:30:43.337080 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.336932 4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="342e684f-1119-4f2e-9f77-0e189842b86c" containerName="istio-proxy" Nov 01 13:30:43.340041 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.340021 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.386495 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.386467 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-podres\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.386622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.386510 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-lib-modules\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.386622 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.386592 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-sys\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.386702 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.386656 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/e44ff208-5e90-40ee-af46-482c893091a3-kube-api-access-vz26j\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.386741 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.386716 4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-proc\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.387822 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.387801 4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9plmq\"/\"default-dockercfg-k7gkk\"" Nov 01 13:30:43.434049 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.434025 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9plmq\"/\"openshift-service-ca.crt\"" Nov 01 13:30:43.434217 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.434103 4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9plmq\"/\"kube-root-ca.crt\"" Nov 01 13:30:43.436392 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.436367 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29"] Nov 01 13:30:43.488184 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488161 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-podres\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488329 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488200 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-lib-modules\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488329 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488235 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-sys\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488329 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488282 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/e44ff208-5e90-40ee-af46-482c893091a3-kube-api-access-vz26j\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488329 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488318 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-podres\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488329 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488325 4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-proc\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488340 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-lib-modules\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488367 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-sys\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.488627 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.488375 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e44ff208-5e90-40ee-af46-482c893091a3-proc\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.540728 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.540695 4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz26j\" (UniqueName: \"kubernetes.io/projected/e44ff208-5e90-40ee-af46-482c893091a3-kube-api-access-vz26j\") pod \"perf-node-gather-daemonset-qxq29\" (UID: \"e44ff208-5e90-40ee-af46-482c893091a3\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:43.652213 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:43.652152 4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:44.014223 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:44.014194 4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29"] Nov 01 13:30:44.017684 ip-10-0-140-174 kubenswrapper[4484]: W1101 13:30:44.017632 4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode44ff208_5e90_40ee_af46_482c893091a3.slice/crio-63b450e47779d957d7b11f899e1a80ecb65399e4b212b256d8366f19891a4d41 WatchSource:0}: Error finding container 63b450e47779d957d7b11f899e1a80ecb65399e4b212b256d8366f19891a4d41: Status 404 returned error can't find the container with id 63b450e47779d957d7b11f899e1a80ecb65399e4b212b256d8366f19891a4d41 Nov 01 13:30:44.553544 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:44.553506 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" event={"ID":"e44ff208-5e90-40ee-af46-482c893091a3","Type":"ContainerStarted","Data":"7c814ea37c281ad36d6bbd568dde82e6ccacca73f3cdf420017356f84ea06b26"} Nov 01 13:30:44.553956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:44.553571 4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" event={"ID":"e44ff208-5e90-40ee-af46-482c893091a3","Type":"ContainerStarted","Data":"63b450e47779d957d7b11f899e1a80ecb65399e4b212b256d8366f19891a4d41"} Nov 01 13:30:44.553956 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:44.553597 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:44.585947 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:44.585920 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wmb4b_3a80816c-07b3-4b1a-9cf5-120d71f4a79b/limitador/0.log" Nov 01 13:30:44.659620 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:44.659561 4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" podStartSLOduration=1.65954723 podStartE2EDuration="1.65954723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 13:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:30:44.657731488 +0000 UTC m=+4114.407674465" watchObservedRunningTime="2026-11-01 13:30:44.65954723 +0000 UTC m=+4114.409490218" Nov 01 13:30:50.568227 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:50.568193 4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-qxq29" Nov 01 13:30:52.592989 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:52.592960 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-jvq5s_ab722746-95b1-45a9-a6f3-efce8f887804/monitoring-plugin/0.log" Nov 01 13:30:55.459094 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:55.459059 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l76dc_3ec9b82f-9d14-4035-a558-d7014e495f23/node-exporter/0.log" Nov 01 13:30:55.862315 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:55.862265 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l76dc_3ec9b82f-9d14-4035-a558-d7014e495f23/kube-rbac-proxy/0.log" Nov 01 13:30:56.267416 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:56.267391 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l76dc_3ec9b82f-9d14-4035-a558-d7014e495f23/init-textfile/0.log" Nov 01 13:30:56.677861 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:56.677775 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5vx6k_cd430b13-1a61-4f19-8139-791ac71c1eff/kube-rbac-proxy-main/0.log" Nov 01 13:30:57.082513 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:57.082468 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5vx6k_cd430b13-1a61-4f19-8139-791ac71c1eff/kube-rbac-proxy-self/0.log" Nov 01 13:30:57.487652 ip-10-0-140-174 kubenswrapper[4484]: I1101 13:30:57.487626 4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5vx6k_cd430b13-1a61-4f19-8139-791ac71c1eff/openshift-state-metrics/0.log" Nov 01 13:31:00.251704 ip-10-0-140-174 kubenswrapper[4484]: E1101 13:31:00.251663 4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5q4xn" containerName="extractor"